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Preface 


This  report  documents  the  SCE  implementation  procedures 
devloped  for  the  Electronic  Systems  Center  (ESC),  Hanscom 
Air  Force  Base.  The  information  contained  in  this  report  was 
developed  before  the  release  of  the  Software  Capability 
Evaluation  Version  1.5  Method  Description  [SCE  93]. 
Inconsistencies  should  be  checked  against  that  document 
which  takes  precedence  in  matters  of  SCE  methodology. 
Similarly,  at  the  request  of  ESC,  this  report  incorporated 
version  1.0  of  the  Capability  Maturity  Model  (CMM)  before 
the  release  of  CMM  version  1.1. 

Abstract  Software  Capability  Evaluation  (SCE)  offers  a  means  to 

evaluate  an  organization's  software  process  capability — that 
is,  how  well  an  organization  manages  the  process  it  uses  to 
create  software.  SCE  provides  a  way  to  compare  an  offeror's 
software  capability  against  a  predefined  standard.  This 
document  is  an  implementation  guide:  it  is  intended  as  a  set 
of  practical  information  which  program  managers  can  use  to 
guide  them  through  the  process  of  using  SCE  in  an 
acquisition. 


Purpose  of  This  The  Software  Engineering  Institute's  (SEI)  Software 
Document  Capability  Evaluation  (SCE)  method,  as  introduced  in  this 

guide,  offers  a  means  to  help  ESC  acquisition  managers 

•  Identify  program  risk  by  evaluating  software  process 
capability  in  source  selection. 

•  Manage  program  risk  by  motivating  contractors  to 
improve  their  software  development  processes  without 
forcing  compliance  to  specific  practices. 
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This  guide  can  be  used  to  implement  the  SCE  method  in  order 
to  achieve  the  goals  above.  It  provides  specific  information 
necessary  to  orchestrate  SCE  use  during  the  source  selection 
process.  Specifically,  this  guide 

•  Provides  guidance  on  how  to  use  the  SCE  method  as  a  tool 
to  identify  software  risk  during  a  source  selection. 

•  Provides  standardized  SCE  implementation  guidance 
which  is  documented,  available  for  review  and  comment, 
and  periodically  modified  as  experience  is  gained  with  its 
use. 

•  Provides  information  which  will  help  acquisition 
organizations  develop  appropriate  policies,  implementing 
instructions,  and  guidelines  to  use  SCE  in  source  selection 
and  institutionalize  SCE  as  a  routine  practice. 

•  Supplements,  but  does  not  replace,  team  training  for 
evaluating  the  software  process  capability  of  contractors. 


The  Software  Capability  Evaluation  method  was  originally 
developed  by  Watts  Humphrey  and  William  Sweet  of  the  SEI, 
with  contributions  from  R.  K.  Edwards,  G.  R.  LaCroix,  M.  F. 
Owens  and  H.  P.  Schultz  of  the  MITRE  Corporation.  On-going 
technical  development,  support  and  validation  of  the  SCE 
method  is  performed  by  the  SETs  Software  Capability 
Evaluation  project. 

Major  contributions  to  the  development  of  this  version  of  the 
guide  have  been  made  by  Captain  Joe  Besselman  (USAF 
government  SEI  affiliate),  Paul  Byrnes,  and  Rick  Barbour 
(SEI).  Reviewers  and  content  builders  included  Edward 
Averill,  Pat  Hanavan,  Bob  Lang,  Cecil  Martin,  Woody  Mead, 
Raj  Puraruk,  Bill  Hefley,  Peter  Malpass,  Mark  Paulk,  and  Jim 
Withey  (SEI),  and  Albert  Johnson  (formerly  of  the  SEI). 

The  SEI  would  like  to  acknowledge  the  external  user 
reviewers  of  the  guide  who  provided  many  valuable 
comments  and  suggestior\s  for  improvement:  Joseph  Billi, 
Martin  Ower\s,  Mario  Olson,  Elizabeth  Gramoy,  David  Rugg, 
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Wooldridge,  Kathie  McCullough,  and  Charlie  Ryan. 
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Introduction 


Intended  The  primary  audience  for  this  document  is  program  office 

Audience  for  personnel  responsible  for  the  softw-are  component  of  an 

This  Document  acquisition.  The  guide  assumes  that  the  program  manager 

will  be  delegated  the  responsibility  for  determining 
appropriate  SCE  usage,  implementation  plarming,  and 
carrying  out  the  SCE  plan.  Program  managers  should  be 
familiar  with  the  material  contained  in  Part  A  of  the  guide,  as 
a  minimum. 

Although  this  document  provides  examples  and  models  of 
SCE  use  in  source  selections  and  talks  to  some  specific  details 
of  source  selection,  it  is  a  guide  and  is  not  intended  to 
compare  to  or  replace  the  use  of  the  AFMC  and  ESC 
regulations  on  source  selection.  In  order  to  work  any  source 
selection  successfully  it  is  impertative  to  work  with  the  local 
procurement  and  legal  staff  as  well  as  the  source  selection 
regulations,  which  take  precedence  over  this  document. 

In  cases  where  SCE  is  ’'eing  used  in  a  source  selection,  source 
selection  personnel  should  be  familiar  with  Parts  A  and  B  of 
this  guide.  Part  A  provides  an  overview  of  the  SCE  method 
and  its  uses.  Part  B  discusses  the  details  of  incorporating  SCE 
into  source  selections  of  specific  acquisitions. 

Structure  Part  A  of  this  guide  provides  an  introduction  to  SCE  that  all 

readers  should  understand.  It  provides  an  overview  of  SCE's 
technical  basis  along  with  a  high  level  view  of  the  mechanics 
of  executing  an  SCE  in  an  acquisition  and  a  discussion  of  the 
underlying  Capability  Maturity  Model  (CMM).  The  emphasis 
in  Part  A  is  on  providing  the  basic  understanding  of  the  SCE 
method  necessary  for  a  user  to  focus  on  the  source  selection 
use  of  the  SCE  method. 

Part  B  focuses  on  how  to  implement  SCE  in  a  source  selection. 
This  portion  of  the  guide  introduces  the  key  activities  and  is 
followed  by  specific  guidance  on  how  to  use  the  method  and 
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the  documentation  needed  to  execute  recommended 
approaches.  The  examples  included  are  modified  instances  of 
an  implementation  of  SCE.  There  may  be  other  approaches 
that  are  equally  valid  and  useful.  Some  audiences  may  only  be 
concerned  with  a  subset  of  the  information  about  the  SCE 
method  contained  in  this  guide;  consequently,  the  materia; 
has  been  orgartized  so  that  different  audiences  can  focus 
attention  on  specific  pieces. 


I 
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Chapter  1  Introducing  SCE 


This  part  of  the  document  introduces  the  Software  Capability 
Evaluation  (SCE)  Method  and  relates  it  to  the  SEI  Capability 
Maturity  Model  (CMM).  Chapter  1  describes  what  SCE  is  and 
Chapter  2  discusses  the  execution  of  SCE  on  an  acquisition. 

Background  Early  attempts  at  improving  the  acquisition  process  began 

with  a  memorandum  published  by  Deputy  Secretary  of 
Defense  Frank  C.  Carlucci  HI  in  1981.  Initiative  11  of  this 
memorandum  (Carlucci  81]  required  the  Department  of 
Defense  (DoD)  to  increase  the  visibility  of  technical  risk  in  the 
budgets  of  acquisition  programs  for  weapon  systems.  In  1986, 
the  United  States  General  Accounting  Office  (GAO)  released 
a  report  titled  "Technical  Risk  Assessment — The  Status  of 
Current  DoD  Efforts"  [GAO  86],  which  examined  the 
methodology  used  for  assessing  technical  risks  within  25 
program  offices.  The  deficiencies  found  by  GAO  prompted 
development  of  various  risk  assessment,  evaluation,  and 
management  publications,  including  the  Defense  Systems 
Management  College  (DSMC)  guide,  "Risk  Management 
Concepts  and  Guidance."  [Deep]  DoD  organizations 
launched  further  initiatives  to  improve  their  risk  assessment 
capability.  One  such  initiative  resulted  in  development  of  the 
Software  Capability  Evaluation  method. 

The  SCE  Method  complements  earlier  work  by  extending 
technical  risk  assessment  to  include  the  software  process.  It 
establishes  software  process  capability  as  a  criterion  for 
source  selection  by  providing  an  orderly  way  to  compare 
offerors'  software  capability  against  a  predefined  process 
maturity  model.  SCE  should  be  used  to  augment  other 
software  risk  assessment  techniques  currently  used  in  source 
selection  and  contract  monitoring. 


CMU/SEI-93-TR-18 


1-1 


Chapter  1 ;  Introducing  SCE 


Purpose  of  SCE 


The  SCE  Method  was  conceived  and  developed  by  the  SEI 
and  the  MITRE  Corporation  in  1987,  at  the  request  of  the  Air 
Force.  It  was  designed  to  help  program  managers  determine 
the  software  process  capability  of  a  contractor  at  one 
organizational  site  (facility  or  location). 

SCE  provides  a  snapshot  of  a  contractor's  past  process 
implementation,  current  process  activities,  and  future  process 
potential.  The  SCE  site  visit  is  an  in-plant  review  conducted 
by  four  to  six  government  personnel  over  a  three  day  period 
at  a  contractor's  facility.  The  output  from  an  SCE  site  visit  is  a 
set  of  findings;  of  strengths,  weaknesses,  and  improvement 
activities  measured  against  the  SEI's  Capability  Mahirity 
Model  (CMM) — ^a  model  which  defines  what  it  means  to  have 
a  mature  software  process  and  why  a  mature  process  results 
in  a  better  product.  The  CMM  is  the  technical  basis  for 
reporting  the  findings  of  the  SCE  Method.  Figure  1-1  lists  the 
documents  which  are  available  through  the  SEI  or  the  Defense 
Technical  Information  Center  (DTIC)  that  describe  the  CMM 
and  the  Maturity  Questionnaire  (MQ). 


Document  Titie 

Report  Number 

Characterizing  the  Software 

Process:  A  Maturity  Framework 

CMU/SEI-87-TR-11 

DTIC  #ADA  182895,  June  1987 

A  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors 

CMU/SEI-87-TR-23 

DTIC#  ADA  187320, 

September  1 987 

Capability  Maturity  Model  for 

Software 

CMU/SEI-91-TR-24 

DTIC  #  ADA  24603,  August  1991 

Key  Practices  of  the  Capability 

Maturity  Model 

CMU/SEI-91-TR-25 

DTIC  #  ADA  24604,  August  1991 

Figure  1-1  Reference  Documents  for  SCE  and  the  CMM 


The  SCE  team  identifies  an  offeror's  strengths,  weaknesses, 
and  any  improvement  activities  in  relation  to  the  CMM  and 
the  sponsoring  organization's  objectives.  The  findings  of  the 
SCE  team  are  incorporated  into  the  source  selection 
sponsoring  organization's  technical /management  team  for 
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The  Capability 
Maturity  Model 

Basic  Concepts 


incorporation  into  acquisition  decisions.  The  SCE  team 
assimilates  data  on  various  project  practices  and  from  these 
creates  an  overall  picture  of  the  organization's  software 
process  capability  relative  to  the  CMM. 

Cost,  schedule,  and  performance  are  high  priorities  for  the 
program  manager.  A  basic  tenet  of  the  SCE  Method  and  CMM 
is  that  the  more  mature  the  contractor's  software  process 
capability  is,  the  more  likely  it  is  that  the  contractor  can  meet 
predicted  cost,  schedule,  and  performance  targets.  Because 
SCE  evaluates  an  organization's  software  process  capability, 
acquisition  organizations  gain  insight  into  this  key  area,  an 
area  not  evaluated  by  the  government  in  the  past.  Note  that 
SCE  supplements,  but  does  not  replace,  the  use  of  other 
considerations  such  as  application  domain  expertise,  past 
performance,  and  organizational  capacity  in  acquisition 
decisions. 


The  Capability  Maturity  Model  (CMM)  is  based  on  the 
premise  that  the  quality  of  a  product  stems  in  large  part  from 
the  quality  of  the  process  used  to  create  it.  To  improve 
product  quality  in  the  Total  Quality  Management  (TQM) 
sense,  the  process  used  for  developing  the  products  should  be 
defined,  understood,  measured,  and  progressively  improved. 
Management  also  has  greater  insight,  understanding,  and 
control  of  risks  as  process  quality  increases.  Concepts  of 
process  and  quality  management  are  applied  to  building  and 
maintaining  software  products  by  using  the  CMM  in 
conjunction  with  the  SCE  Method. 

Capability  Maturity  Model  for  Software  [Paulk  91a]  describes  the 
framework  of  the  CMM  (Vl.O).  The  CMM  organizes  common, 
proven  software  development  practices  into  a  structured 
framework  which  can  be  used  to  focus  quality  improvement 
efforts.  Use  of  the  CMM  as  described  in  the  above  report  will 
begin  November  1992.  SCE  teams  will  be  taught  to  evaluate 
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The  Maturity 
Framework  and 
Software  Quality 
Improvement 


contractors  using  the  CMM  version  1.0.  Previously,  SCE  teams 
were  trained  using  the  maturity  framework  as  described  in 
Characterizing  the  Software  Process:  A  Maturity  Framework 
[Humphrey  87a]  and  A  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors-,  [Humphrey  87b]  referred 
to  in  this  document  as  CMM  version  0. 

The  maturity  model  discussed  in  Capability  Maturity  Model  For 
Software  [Paulk  91a]  consists  of  five  maturity  levels  with  key 
process  areas  (KPAs)  assigned  to  each.  Figure  1-2  shows  the 
name  and  number  of  each  level  in  the  left  column.  Level  1  is  the 
lowest;  Five  is  the  highest.  The  general  characteristics  of  an 
organization  functioning  at  a  partioilar  maturity  level  are 
listed  in  the  middle  column,  and  the  KPAs  associated  with 
each  level  of  the  maturity  model  are  on  the  right.  This  maturity 
model  is  the  foundation  upon  which  the  CMM  was  built.  A 
reading  of  Key  Practices  of  the  Capability  Maturity  Model  [Weber 
91]  will  reveal  an  expansion  of  the  CMM  into  more  practical 
aspects  of  software  engineering. 
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Level 

Characteriatic 

Key  Process  Area 

Optimizing  (5) 

•  Continuous  process 
improvement  capability 

•  Process  change  management 

•  Technology  innovation 

•  Defect  prevention 

Managed  (4) 

•  Product  quality  planning 
and  tracking  of  mea¬ 
sured  software  process 

•  Process  measurement  and 
analysis 

•  Quality  management 

DeHned  (3) 

•  Life  cycle  process 
defined  and  institutional¬ 
ized  to  provide  product 
quality  control 

•  Peer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 

•  Integrated  software  manage¬ 
ment 

•  Training  program 

•  Organization  process  defini¬ 
tion 

•  Organization  process  focus 

Repeatable  (2) 

•  Management  oversight 
and  tracking  of  project 

•  Stable  planning 

•  Software  Configuration  Man- 
agetTwnt 

•  Software  quality  assurance 

•  Software  subcontract  manage¬ 
ment 

•  Software  project  tracking  and 
oversight 

•  Software  project  planning 

•  Requirements  management 

inftial(l) 

•  Ad  hoc  (unpredictable, 
chaotic) 

•  “People’ 

Figure  ^~2  Capability  Maturity  Model  Levels  and  Key  Pro¬ 
cess  Areas 

The  KPAs  are  "stepping  stones"  for  moving  to  higher  levels — 
that  is,  a  company  must  be  proficient  in  the  KPAs  within  a 
maturity  level  in  order  to  move  up  to  the  next  level.  For 
instance,  without  a  stable  and  rep)eatable  software  project 
planning  system,  investments  in  a  formal  definition  of  the 
organization's  tedmical  software  process  aren't  likely  to 
overcome  the  limitations  imposed  by  poor  software  project 
planning.  The  model  is  organized  with  basic  project 
management  practices  at  the  lowest  levels,  and  the  more 
sophisticated  practices  supporting  product  development  at 
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the  higher  levels.  While  moving  up  to  higher  levels,  a 
company  must  improve  as  well  as  maintain  proficiency  in  the 
KPAs  of  the  lower  levels. 

The  model  is  structured  to  demortstrate  that  the  greatest 
benefit  from  quantitative  measures  of  the  process  come  when 
all  of  the  KPAs  through  the  defined  level  are  successfully 
implemented  first.  The  CMM  in  effect  provides  a  step  ladder 
which  management  can  use  to  prioritize  scarce  resources 
towards  the  greatest  long  term  benefit  to  the  organization. 
With  this  structure  it  is  predicted  that  an  organization  can 
better  understand  the  processes  implemented  throughout  the 
company,  and  design  an  improvement  plan  with  better 
chances  of  success.  Subsequently,  this  understanding  leads  to 
more  efficient  and  effective  investments  in  people,  process, 
and  technology  which  are  the  key  elements  of  a  sound 
organizational  improvement  program. 


A  General 
Description  Of 
Organizations  At 
Each  Maturity  Level 


Five  basic  levels  of  process  maturity  have  been  defined  in  the 
model  to  desaibe  the  progression  from  an  ad  hoc  software 
process  to  one  that  is  under  statistical  control  and  can  act  as  a 
foundation  for  continuous  process  improvement. 

Level  1  (initial):  Projects  can  be  characterized  as  routinely 
being  late  and  exceeding  the  planned  budget. 

Level  2  (repeatable):  The  organization  installs  basic 
management  controls  and  generally  learns  to  manage  its  costs 
and  schedules  while  building  similar  software  products.  The 
focus  is  on  the  product,  and  the  management  system  is  largely 
reactive. 

Level  3  (defined):  The  process  is  understood  and  explicitly 
defined.  Here,  the  organization  introduces  a  structured 
framework  for  software  development  and  establishes 
dedicated  process  improvement  resources. 
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Level  4  (managed):  the  processes  are  quantified,  measured, 
and  reasonably  well  controlled.  Data  is  available  to  establish 
improvement  priorities  and  to  support  tool  and  technology 
investment.  In  statistical  process  control  terms,  special  causes 
of  process  variation  are  under  control. 

Level  5  (optimizing):  The  common  causes  behind  process 
variations  are  systematically  addressed,  and  process  data  are 
used  to  improve  the  process  in  respor\se  to  new  and  evolving 
issues  and  capabilities.  The  organization  focuses  on 
continuous  improvement  guided  by  process  data. 


The  Model-Based 
Structure  of  the 
CMM 


Figure  1-3  shows  how  the  various  aspects  of  the  CMM  relate 
to  one  another.  Maturity  levels  are  the  highest  abstraction, 
while  the  key  practices  are  the  most  detailed  portion  of  the 
model.  The  key  indicators,  and  hence  the  maturity 
questionnaire,  are  derived  from  the  key  practices. 
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Inherent  ability  of 
next  project  to 
meet  quality, 
schedule,  and  cost 
goals 


Ability  of  a  process 
to  control  or  avoid 
significant  causes 
of  poor  quality, 
cost,  and  schedule 
performance 


Organization 
process  maturity 
required  for  next 
maturity  level 


The  software 
processes  that 
are  main 
contributors  to 
maturity  level 
process  capability 

Principal  software 
practices  for 
achieving 
effective, 
reproducible 
process  results 


The  practices, 
expressed  as 
questions,  that 
have  the  highest 
reliability  in 
indicating  that  a 
process  area  is 
satisfied 


Effects  of  Increasing 
Maturity  On 
Program 
Predictability 


Figure  1*3  Capability  Maturity  Model  Structure  (CMM  V1 .0) 

Another  important  premise  of  the  CMM  is  that  increasing 
process  maturity  leads  to  reduced  variance  in  the 
performance  of  the  software  process  and  increased  software 
process  capability,  as  illustrated  in  Figure  1-4.  Organizations 
at  the  initial  maturity  level  will  miss  the  target  performance 
on  most  of  their  projects.  Occasionally,  a  strong  manager  may 
drive  one  project  in  a  Level  1  organization  to  a  higher  level  of 
process  capability,  but  that  capability  is  not  present 
throughout  the  organization.  As  organizations  reach  higher 
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maturity  levels,  there  is  a  more  effective  infrastructure  in 
place  to  drive  the  process  capability  applied  on  the  next 
acquisition  to  achieve  program  goals. 


With  increasing  maturity: 

•  accuracy  increases 

•  variance  reduces 

•  target  improves 


Levels 


Level  2 


Figure  1-4  Predicted  Distribution  of  Performance  as 
Organizational  Process  Maturity  Increases 

The  defined  level  is  the  starting  point  from  which  to  achieve 
statistical  process  control,  which  enables  an  organization  to 
understand  where  and  how  the  quality  and  productivity  of  a 
process  can  be  continuously  improved.  Level  5  (optimizing)  is 
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Software  Process 
Areas  Covered  in 
the  CMM 


the  level  at  which  data  is  available  to  tune  the  process  itself. 
The  optimizing  level  is  not  intended  to  be  an  end-point, 
however,  and  should  properly  be  viewed  as  a  beginning  of 
orderly  and  managed  process  improvement.  The 
evolutionary  journey  from  Level  1  to  Level  5  can  then  be  seen 
as  building  the  organizational  capability  to  make  sustained 
and  continuous  improvement. 


The  SCE  team  determines  what  process  activities  are  actually 
implemented  within  an  offeror's  organization.  Figure  1-5 
depicts  the  separate  software  process  areas  defined  in  the 
Maturity  Framework  and  expanded  in  the  CMM.  'The  shaded 
boxes  reflect  areas  covered  by  the  CMM  key  process  areas 
(KPAs). 
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Figure  1*5  Level  2  and  3  Key  Process  Areas  in  the  CMM 

Figure  1-5  illustrates  a  key  difference  between  traditional 
views  of  software  development  process  and  the  Maturity 
Framework/ CMM.  The  activities  within  the  dashed  rectangle 
are  essentially  product  building  activities  (e.g.  Requirements 
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Analysis,  Design,  Testing).  Traditionally  software  product 
development  has  focused  on  product  building  activities  such 
as  these.  But  this  traditional  approach  ignores  many  key 
process  activities  (shown  as  shaded  boxes)  which  are  crucial  to 
successful  product  building. 

The  SCE  team  examines  not  only  the  part  of  the  software 
development  process  that  typically  shows  up  in  the  Software 
Development  Plan  (dashed  rectangle  depicted  as  software 
product  development  activities),  but  also  the  forces  acting  on 
the  process  (dashed  Lines)  and  the  relationships  between  the 
forces. 
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The  Process 

Perspective  vs. 

Traditional 

Product- 

Oriented 

Perspectives 


Figure  1-6  shows  the  DoD-STD-2167A  software  development 
activities  associated  with  the  typical  Engineering 
Manufacturing  Development  (EMD)  program.  Most 
programs  throughout  the  system  acquisition  lifecycle  are 
managed  by  a  lifecycle  development  model  that  is  composed 
of  these  activities. 


System 

Req. 

Analysis/ 

Software 

Design 

Req. 

Analysis 

Preliminary 

Design 

Detailed 

Design 

Coding  and 

CSU 

CSC 

Testing 

Integration 

. 

Testing 

CSCI 

Testing 

SRR 

SDR  SRS 

PDR 

CDR 

System 
Integration 
and  Test 


TRR 


SRR  -  System  Requirements  Review 

SDR  -  System  Design  Review 

SRS  -  Software  Requirement  Specification 

PDR  -  Preliminary  Design  Review 

CSU  -  Computer  Software  Unit 

CSC  -  Computer  Software  Component 


CSCI  -  Computer  Software  Configuration  Item 
COR  -  Critical  Design  Review 
TRR  -  Test  Readiness  Review 
FCA-  Functional  Configuration  Audit 
PCA  -  Physical  Configuration  Audit 


FCA/ 

PCA 


Figure  1-6  DoD-STD-2167A  Activities 


Each  of  these  development  steps  are  made  more  effective  by 
support  from  the  key  process  areas  (KPAs)  in  the  CMM. 
Historically,  government  programs  have  focused  on  the 
development  of  software  products  without  regarding  the 
supporting  processes  as  equally  important.  DoD-STD-2167A 
reflects  this  bias  toward  product  delivery.  The  KPAs  provide 
a  supporting  process  environment  in  which  the  organization 
can  make  effective  decisions  regarding  the  deliverables 
shown  in  Figure  1-6-  Thus  a  focus  on  KPAs  balances  the 
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historical  product  orientation  with  a  process  focus  which  is 
critical  in  predicting  contractor  performance  and  measuring 
product  development  expertise. 


Operational 
needs  — 


System  Requirements 
Process 


SSDD 


Interface  Requirements 
Process 


IRS  product 


SSDD 


Software  Requirements 
Process 


Software  Requirements 
Engineering  and 
Preliminary  Design 


SRS  products 
1  per  CSCi 


Process 


Software  Test  Plan 
DI-MCCR-800UA 


SDD  product 
preliminary 
■aesign 


Detailed 

Software  Design 


Process 


SDD  detailed  _ 

design  level  (CSUs) 


f 


SystemfSegment 
Design  Document  (SSDD) 


Software  Requirements 
Specification  (SRS) 

Spec,  of  required  software 
capabilities 


Software  Design 
Document  (SDD) 
Spec,  of  software’s 
preliminary  design 


Specification  of 
'  software’s  detailed  design 


The  sum  of  the  activities  to 
product  deliverables  does  not{ 
equal  the  process 


Figure  1<7  A  Product  Approach  to  Software  Process 

Figure  1-7  graphically  depicts  a  product  documentation 
approach  to  software  process  often  adopted  by  acquisition 
and  contractor  management.  One  common  problem  is  that 
people  often  equate  the  organization's  software  process  to  the 
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creation  of  output  products  during  each  of  the  steps  shown 
above.  The  actual  software  development  process 
implemented  in  an  organization  contains  many  more 
activities  thain  the  steps  directly  related  to  the  product 
building  parts  of  the  process.  Documents  defining  specific 
intermediate  products  are  not  the  process,  but  are  in  fact 
artifacts  of  the  process  which  is  implemented  in  an 
orgaruzation.  A  successful  program  manager  will  focus  on 
process  (as  well  as  product)  as  a  predictor  of  future 
performance  of  the  development  of  the  product  he  or  she  is 
tasked  to  acquire. 

Another  problem  with  the  product  view  of  software  process 
shown  in  Figure  1-7  is  the  fact  that  products  built  by  a 
software  development  organization  are  often  overcome  by 
technological  or  environmental  obsolescence  very  quickly.  In 
this  acquisition  environment,  it  is  difficult  to  estimate 
accxorately  new  program  risks  by  investigating  existing 
systems  because  the  existing  systems  may  be  obsolete. 

New  program  risk  assessment  is  difficult  because  one  cannot 
assume  an  organization  has  the  ability  to  take  on  a  more 
technologically  advanced  or  larger  project,  make  effective  use 
of  new  technology,  or  address  changes  in  the  threat  or 
development  environment  based  on  past  project  success 
alone.  This  is  because  software  system  requirements  are 
rarely  the  same  from  project  to  project.  New  requirements 
and  advances  in  technology  inevitably  mean  that  old  ways  of 
conducting  business  will  be  inadequate  unless  the 
organization  changes  its  ways.  This  requires  a  focus  on  the 
management  process  which  supports  the  product 
development  activities. 

Using  prior  data  to  evaluate  parameters  of  the  new  system 
assumes  the  new  system  is  'precedented,'  meaning  it  is 
similar  to  systems  previously  built.  According  to  the  Air  Force 
Studies  Board  [AFSB  89],  a  precedented  system  meets  three 
criteria: 

1.  A  stable  set  of  software  requirements  exists  that  is  not 
substantially  different  from  that  of  a  previous  system. 
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2.  The  digital  system  architecture  and  software  design  that 
will  satisfy  the  given  requirements  are  known. 

3.  The  contractor's  system  engineering  and  software  teams 
communicate  effectively  with  each  other  and  have  prior 
experience  in  developing  a  similar  system. 

According  to  the  AFSB,  the  majority  of  USAF  systems  (and  by 
implication  all  modem  complex  weapon  systems)  can  be 
considered  unprecedented  in  that  major  portions  of  the 
software  development  do  not  meet  at  least  one  of  the  criteria 
above. 

The  ability  to  address  these  issues  is  as  much  dependent  on 
the  organization's  software  process  capability  as  it  is  on  past 
product  building  successes.  Risk  abatement  can  be  better 
achieved  by  evaluating  software  process  capability  in 
addition  to  using  traditional  product  specific  methods. 
Institutionalization  of  the  key  software  practices  provides 
another  indicator  of  how  the  organization  is  likely  to  address 
technological  challenges  and  manage  risks, 

A  program  manager  must  wrestle  with  product-oriented 
questions,  but  answers  to  these  questions  will  not  address 
many  of  the  software  process  specific  risks  in  acquisitions. 
Typical  capacity  reviews  derive  current  capability  from  past 
performance,  and  are  used  to  answer  many  product-oriented 
questions.  The  SCE  Method  adds  value  to  the  typical  capacity 
review  because  it  measures  a  subset  of  software  process 
attributes  which  are  believed  to  indicate  organizational  ability 
to  successful!)'  develop  the  product  to  be  acquired — in  other 
words,  to  take  on  future  challenges,  rather  than  past  efforts. 

An  example  which  illustrates  the  difference  between  a 
capacity  review  and  a  SCE  is  in  the  training  area.  Assume  a 
new  procurement  calls  for  500  KSLCXZ  in  Ada,  and  50  Ada 
programmers  are  expected  over  the  life  of  the  EMD  phase.  In 
a  capacity  review,  the  government  team  is  primarily 
concerned  with  whether  the  contractor  has  enough 
experienced  Ada  people  on  hand  to  perform  the  work.  In  an 
SCE,  the  emphasis  is  not  on  specific  people,  but  on  the  process 
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Contractor 
Attributes 
Examined  by 
SCE 


the  contractor  uses  for  training — in  this  case,  the  process  and 
plans  for  training  the  Ada  programmers  if  not  enough  are 
available  to  work  on  the  project. 

Both  process  and  product  perspectives  are  important  to  the  program 
manager.  The  contractor's  software  process  is  by  no  means  a 
program  manager's  only  concern.  But  if  the  program  manager 
focuses  on  process  capability  in  addition  to  traditional 
product  oriented  concerns,  risk  areas  can  be  identified  up 
front,  and  quality  problems  which  may  occur  on  the  program 
due  to  those  risks  can  be  proactively  managed  and  prevented. 

The  findings  from  a  SCE — strengths,  weaknesses  and 
improvement  activities  relative  to  the  CMM — can  influence 
the  source  selection  decision  or  contribute  to  the  future 
direction  of  an  on-going  program.  However,  there  are  many 
other  attributes  besides  the  software  process  that  are 
important  in  determining  the  most  qualified  contractor  to  do 
a  job. 
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Contractor 

•  Manufacturmg 

•  Tools  and  Instrumentation 

Attributes  Which 

«  Conner  Security 

•  Pattern 

Could  Be 

•  Signal 

Recognition 

Considered  in 

Processing 

*  Hardware 

a  Software*lntensive 

•  Object-Oiiented 

Enj^neering 

Acquisition 

Programming 

•  Mabita^iabiiity 

•  System  Engineering 

•  Ada  Expertise 

«  Communications 

•  Prior  Performance 

•  Software  Safety 

•  Peopia 

Analysis 

•  Sto  Capacity 

•  Formal  Methods 

•  Envkonments 

•  Software  Tasting 

•  Location 

«  Human 

•  ReliabHity 

En^neering 

•  Hardware 

•  Facilities 

Ei^neering 

«  Reuse 

•  Networks 

•  Requirements  Manage- 

•  Integrated  Software 

ment 

Management 

•  Software  Project  Planning 

•  Software  Product  Engi- 

•  Software  Project  Track- 

neering 

ing  and  Oversight 

•  intergroup  Coordination 

•  Software  Subcontract 

•  Peer  Reviews 

Management 

•  Process  Measurement 

Key  Process  Areas 

•  Software  Quality  Assur- 

and  Analysis 

Considered  by  SCE 

ance 

•  Quality  Management 

•  Software  Configuration 

•  Defect  Prevention 

Management 

•  Technology  Innovation 

•  Organization  Process 

•  Process  Change  Man- 

Focus 

•  Organization  Process 
Definition 

agement 

Figure  1-8  Sample  of  Contractor  Attributes  That  Could  Be 
Considered  in  a  Software-Intensive  Acquisition 

Figure  1-8  depicts  the  software  intensive  attributes  of  a 
contractor's  organizational  site  and  those  considered  by  SCE. 
A  SCE  site  visit  to  a  contractor's  facility  is  an  intense 
examination  of  the  organization  that  reveals  its  software 
development  process  capability  and  improvement  activities. 
All  of  the  attributes  shown  above  are  important  to  an 
acquisition,  but  only  those  related  to  software  process  are 
captured  and  recorded  in  the  findings  during  the  SCE  site 
visit 
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The  Resuits  of 
an  SCE  Site  Visit 


Areas  other  than  software  process — tools,  systems 
engineering,  and  skill  and  experience  with  a  particular 
language — should  be  investigated  when  performing  a  source 
selection.  Under  no  circumstances,  however,  should  these 
additional  review  areas  contribute  to  a  site's  SCE  findings. 
They  should  be  investigated  outside  the  structure  of  the  SCE 
site  visit  to  ensure  repeatability,  consistency,  and  reliability 
between  teams  conducting  site  visits  at  other  contractors.  This 
is  important  to  ensure  fairness  and  equitable  treatment  of  all 
competing  offerors  during  a  source  selection.  While  the  SEI 
supports  the  need  to  review  other  critical  areas  which  are  not 
covered  in  the  CMM  KPAs,  no  attempt  to  merge  these  areas 
into  the  SCE  findings  should  be  made  on  site.  The  current 
method  of  teaching  and  conducting  a  SCE  site  visit  follows 
the  decomposition  of  the  CMM  along  the  lines  of  the  key 
process  areas  (KPAs). 

Findings  of  strengths,  weaknesses,  and  improvement 
activities  against  the  CMM  KPAs  are  the  result  of  a  SCE  site 
visit  these  are  used  within  the  government  acquisition 
process.  The  site  visit  process  is  essentially  a  data  collection 
activity  which  extends  insight  into  an  area  which  has  been 
lacking  in  the  past,  an  offeror's  software  process  capability. 
Figure  1-9  shows  a  sample  of  a  detailed  finding  for  the 
Software  Project  Planning  KPA. 


CMU/SEI-93-TR-18 


1-19 


Comparing  SPA 
and  SCE 


Chapter  1 ;  Introducing  SCE 


Software  Project  Planning 

Commitment  process  at  senior  and  first-line  management  levels 
requires  strengthening 

Strengths 

•  Project  responsibilities  are  defined  and  documented 

•  Mechanism  in  place  to  assure  that  software  estimates  are  tracked 
In  relation  to  software  activities 

Weaknesses 

•  Commitment  procedures  at  senior  and  first-line  management  lev¬ 
els  could  not  be  validated  by  the  team 


Improvement  Activities 

Task  group  is  in  place  to  address  senior  management  issue 


Figure  1  >9  Sample  Key  Process  Area  Finding 

The  SEI  has  developed  Software  Process  Assessments  (SPA) 
and  SCE  as  part  of  a  strategy  to  improve  the  state  of  the 
practice  in  software  engineering.  Both  methods  address  the 
SEI  vision  of  supporting  the  evolution  of  software 
engineering  from  an  ad  hoc,  labor-intensive  activity  to  a 
managed,  technology-supported  engineering  discipline. 

Differences  in  the  methods  stem  from  the  motivations, 
objectives,  outcomes,  and  ownership  of  findings.  These 
factors  lead  to  substantive  differences  in  interview  dynamics, 
scope  of  inquiry,  information  being  gathered,  and 
formulation  of  the  outcome  (findings). 
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Figure  1-10  highlights  several  of  the  major  differences 
between  the  two  methods  which  affect  the  way  they  are 
performed. 


Amcm  merit 

Evaluation 

Used  by  organization  to  improve  its 
software  processes 

Used  by  OoO  in  source  selection  and 
contract  monitoring 

Assesses  process  practice 

Substantiates  current  practice 

Acts  as  catalyst  for  process 
improvement 

Evaluates  contractor’s  commitment  to 
improve 

Provides  input  for  improvement 
action  plan 

Analyzes  contract  performance 
potential 

Findings  may  include  issues  not 
explicit  in  the  maturity  model 

Findings  restricted  to  maturity  nx>dei 
issues 

Collaborative:  members  of 
organization  must  be  on  team 

OoO-oriented:  members  of 
organization  not  on  team 

Applies  to  organization,  not  individual 
projects,  contracts 

Organizational  data  but  applied  to  a 
specific  contract  award 

Input  for  improvement  action  plan  to 
unfreeze  organization. 

input  for  award  decision,  contract 
monitoring,  or  risk  management 

Figure  1-10  Differences  Between  Process  Assessments  and 
Capability  Evaluations 

Critical  differences  an  SCE  user  must  vmderstand  are  the  basic 
assumptions  built  into  the  methods  themselves.  First,  the 
current  SPA  method  assumes  that  members  of  the 
organization  will  cooperate  openly,  fully,  and  in  a 
collaborative  manner  which  provides  truthful  contributions. 
This  assumption  is  made  because  presumably  participants 
have  no  reason  to  mislead  SPA  team  members,  who  are  from 
their  own  organization,  and  are  dedicated  to  improving  the 
organization  with  the  full  support  and  commitment  from 
senior  management.  The  SPA  findings  are  intended  to  be 
incorporated  into  action  plans  for  organizational 
improvement,  and  therefore  do  not  attempt  to  verify,  by 
examination  of  documentation,  every  assertion  made  during 
the  interview  process.  SCE  assumes  that  members  of  the 
evaluated  organization  will  make  every  attempt  to  put  their 
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organization's  software  process  capability  in  the  best  possible 
light.  This  is  because  their  company's  livelihood  and  therefore 
their  own  careers  and  livelihoods  may  be  at  stake  since  the 
SCE  findings  are  used  as  factors  in  determining  potential 
monetary  awards  to  the  company.  Thus,  every  attempt  is 
made  by  the  SCE  team  to  verify  facts  through  use  of  a 
document  review  process. 

An  SCE  team  consists  of  one  government  team  leader  and 
three  to  five  government  personnel  and/ or  their  engineering 
support  contractor  personnel.  This  team  may  include  a 
procurement  member  or  observer.  The  SPA  team  consists  of 
six  to  eight  members  either  entirely  from  the  organization 
itself,  or  from  both  the  organization  and  either  the  SEI  or  one 
of  the  SETs  licensed  SPA  vendors. 

Finally,  SPAs  are  longer  in  duration  (five  days),  involve  more 
people  from  the  assessed  organization,  and  are  generally 
more  costly  than  evaluations  (three  days  duration).  SPA  data 
is  not  recommended  for  use  on  government  source  selections 
because  of  the  inherent  differences  between  the  two  methods 
as  explained  above.  Little  information  from  SPAs  can  be 
directly  applied  by  an  acquiring  organization  for  the  purpose 
of  selecting  the  most  appropriate  offeror. 

The  methods  are  similar  in  that  they  both  use  the  framework 
of  the  CMM  to  structure  a  detailed  investigation  into  software 
practices,  and  both  require  extensive  training  and  experience 
to  conduct  properly.  SPA  training  does  not  prepare  a  team  to 
perform  SCEs,  and  SCE  training  does  not  prepare  a  team  to 
conduct  SPAs. 

Both  the  SPA  and  SCE  methods  seek  to  identify  the 
organization's  software  process  strengths  and  weaknesses  as 
measured  against  the  KPA  goals  of  the  CMM.  Both  seek  to 
motivate  the  contractor  to  embody  the  major  weaknesses  in 
an  aggressive  software  process  improvement  plan.  However, 
the  source  of  the  improvement  motivation  is  clearly  different. 
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The  findings  from  an  SCE  deal  strictly  with  organizational 
strengths  and  weaknesses  against  the  CMM,  not 
recommendations  to  rectify  the  problems  as  is  the  case  with 
SPA. 

The  differences  between  the  two  methods  are  important  to 
acquisition  managers.  It  is  important  because  the  government 
may  benefit  in  the  long  run  by  conducting  business  with 
contractors  that  are  implementing  software  process 
improvements.  Although  SCE  use  may  focus  the  attention  of 
senior  executives  of  hesitant  contractors  on  investments  in 
software  process  improvement,  the  greatest  benefits  will  be 
apparent  to  those  firms  who  have  embraced  the  concepts  of 
process  improvement  without  government  pressures  or 
incentives.  Acquisition  managers  must  understand  the 
limitations  posed  by  SCE  and  SPA,  and  how  they  relate  to 
their  acquisition. 


Facilitating  SCE 
on  Your 
Program 


Integrating  the  SCE  Method  into  an  acquisition  involves  four 
steps: 

1.  Identifying  the  maturity  of  the  contractor's  current 
software  process  in  terms  of  strengths,  weaknesses,  and 
any  improvement  activities  relative  to  the  CMM  KPAs. 

2.  Assessing  program  risk  and  how  the  contractor's  im¬ 
provement  plans  alleviate  that  risk. 

3.  Making  continuous  process  improvement  a  part  of  the 
contractual  relationship  with  the  contractor. 

4.  Monitoring  software  process  performance  and  developing 
a  working  relationship  with  the  contractor  (as  opposed  to 
an  adversarial  relationship). 

The  SEI  provides  briefings  and  presentations  to  acquaint 
acquisition  agencies  with  the  evaluation  method.  If  an 
acquisition  agency  decides  to  use  the  evaluation  method,  it 
sends  representatives  to  the  SEI  training  courses;  SCE 
Overview  Seminar  and  SCE  Team  Training.  The  overview 
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helps  acquisition  agencies  realize  the  implications  and 
benefits  of  using  software  capability  evaluations,  and  the 
team  training  helps  acquisition  teams  prepare  to  conduct  SCE 
site  visits.  They  can  then  conduct  a  pilot  use  of  SCE.  After 
completing  several  pilots,  the  acquisition  agency  can  discuss 
its  experiences  with  the  SEI  and  decide  whether  to  install  the 
evaluation  method  on  a  broader  scale.  The  timeline  for 
transferring  the  evaluation  method  into  an  acquisition 
process  is  outlined  in  Figure  1-11. 
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0  ^ 

Contact 

Presentations,  briefings,  papers 

Awareness 

SEI  Course:  SCE  Overview 

Executive  commitment  and  funding 

6  - 

Plan  for  pilot  use  (Selected  programs) 

SCE  team  selection 

12  - 

- 

Understanding 

SEI  Course:  Evaluation  Team  Training 

SCE  pilot  use  (Selected  programs) 

Evaluation  of  SCE  Method 

- 

Feedback  to  SEI 

Plan  for  installing  SCE 

18  - 

■ 

Installation 

Tailor  SCE  products 

- 

Use  SCE  and  provide  feedback  to  SEI 

24  - 

- 

Adoption 

Draft  and  improve  policy  on  SCE  use 

Revise  and  approve  policy  on  SCE  use 

Develop  an  institutional  SCE  capability 

30  - 

- 

Institution¬ 

alization 

SEI  Course:  Train  the  SCE  trainers 

SCE  use  is  organizational  practice 

Figure  1-11  Steps  for  Transferring  the  Evaluation  Method 
into  an  Organization 


Incorporating  SCE  into  an  acquisition  begins  with  reading 
Chapters  1  and  2  of  this  guide.  Once  the  model  is  understood, 
the  different  applications  of  the  model  are  recognized,  and  it 
is  understood  what  SCE  can  provide,  a  program  office  should 
begin,  at  a  macro  level,  completing  the  activities  of  the 
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checklist  found  in  Figure  1-12.  Remember,  facilitating  SCE  on 
most  acquisitions  is  not  going  to  be  as  easy  as  following  a 
checklist. 


Planning  The  implementation  of  Software  Capability 
Evaluation 


□  Attend  SEI  Course:  SCE  Overview 

□  Selected  engineering  staff  should  read  remainder  of  guide 

□  Determine  program  use 

□  Include  SCE  Method  in  acquisition  planning  documents  and 
request  for  proposal 

O  Select,  register,  and  train  SCE  team 

□  Conduct  SCE,  save  results,  and  capture  lessons  learned 

□  Brief  the  SEI  on  use  of  SCE 


Figure  1-12  SCE  Implementation  Checklist 

The  SCE  Method  is  flexible  because  the  application  of  the 
method  can  be  tailored  without  changing  the  conduct  of  the 
site  visit.  Thus,  different  types  of  acquisitions,  types  of 
processes,  size  of  an  organization,  and  degrees  of  process 
automation  or  use  of  tools  can  be  accommodated  by  the 
method.  This  reflects  the  experience  that  in  most  government 
contexts  the  acquisition  process  will  differ,  within  the 
constraints  expounded  in  the  Federal  Acquisition  Regulation 
(FAR).  It  is  acceptable  to  have  alternative  approaches  to 
implementing  SCE  within  a  specific  context.  The  intent, 
however,  is  to  have  the  site  visit  itself  be  identical,  a  "black 
box"  from  site  to  site.  In  other  words.  Army  Material 
Command  (AMC)  may  do  source  selections  differently  from 
Air  Force  Electronic  Systems  Center  (ESC)  and  the  Naval 
Command,  Control  and  Ocean  Surveillance  Center 
(NCCOSC),  RDT&E  Division  (NRAD),  but  all  should  do  SCE 
site  visits  the  same  way.  This  ensures  flexibility  in  the  use  of 
the  method  while  ensuring  reliability,  consistency,  and 
repeatability  of  the  evaluation  method  itself. 
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Those  responsible  for  implementing  SCE  in  a  source  selection 
are  the  following: 

The  Source  Selection  Authority  (SSA),  who  is  responsible  for 
the  efficient  and  proper  conduct  of  the  source  selection 
process. 

The  Source  Selection  Advisory  Council  (SSAC),  which  ranks 
offeror  proposals  according  to  an  evaluation  plan. 

The  Source  Selection  Evaluation  Board  (SSEB),  which 
evaluates  offeror  proposals. 

The  Source  Selection  Evaluation  Team  (SSET)  is  a  combined 
SSEB  and  SSAC.  They  perform  the  responsibilities  set  forth  in 
APR  70-15  and  APR  70-30.  ESC  uses  either  an  SSEB  or  SSET  on 
their  source  selections,  from  here  on  the  use  of  SSEB/T  or 
SSEB  will  refer  to  either  SSEB  or  SSET,  whichever  structure  is 
being  used. 

The  Procuring  Contracting  Officer  (PCO),  who  is  responsible 
for  solicitations  and  contracts,  communications  with  offerors, 
consistency  of  the  source  selection  plan  with  the  Federal 
Acquisition  Regulation  (FAR),  contract  award,  and  other 
requirements  and  functions  specified  in  the  FAR  except  the 
source  selection  responsibilities  of  the  SSA. 

The  Program  Manager  (PM),  who  is  responsible  for 
developing  and  implementing  the  acquisition  strategy, 
preparing  the  source  selection  plan,  and  obtaining  SSA 
approval  of  the  plan  before  issuance  of  the  Request  For 
Proposal. 

The  Software  Capability  Evaluation  team  is  chaired  by  the 
goverrunent  and  consists  of  a  group  of  four  to  six 
appropriately  experienced  (e.g.  education,  domain  expertise, 
numbers  of  years  experience)  personnel  who  are  trained  in 
applying  the  SCE  Method.  The  SCE  team  is  responsible  for 
applying  the  SCE  Method,  including  preparing  for  and 
conducting  the  site  visits  and  reporting  the  findings. 

Care  in  implementing  SCE  is  important.  The  SCE  team  must 
be  properly  selected  and  trained  to  prepare  the  team  for  the 
site  visit.  Only  experienced  and  trained  teams  can  us  the  SCE 
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Method  in  the  intended  manner.  Training  consists  of  two 
courses.  The  first.  Software  Capability  Evaluation  Overview 
Seminar,  provides  a  briefing  on  the  concepts,  benefits,  and 
guidelines  and  logistics  of  SCE  to  the  acquisition  management 
team.  The  second,  SCE  Team  Training,  provides  a  review  of 
software  process  capability  relative  to  the  CMM,  how  to 
conduct  an  SCE  site  visit,  and  team  development  activities 
including  case  studies  for  SCE  team  members. 
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Chapter  2  Preparing  to  Use  SCE 


The  on-site  review  portion  of  an  SCE  is  the  three-day  visit 
(exclusive  of  travel)  to  a  contractor's  facility  made  by  the 
program  office's  SCE  team  in  order  to  determine  the 
contractor's  strengths,  weaknesses,  and  improvement 
activities  against  the  CMM.  Some  of  the  steps  required  to 
prepare  for  an  SCE  are  specific  to  its  use.  A  project  officer,  or 
alternatively  a  SCE  team  leader  should  prepare  for  a  SCE  by: 

•  Determining  SCE  use. 

•  Selecting  the  SCE  team. 

•  Training  the  SCE  team. 

•  Providing  directions  to  the  contractor,  through  the  RFP 
and  the  PCO. 

•  Screening  contractor  responses. 

•  Preparing  on-site  materials. 

•  Coordinating  on-site  activities. 

The  timeline  in  Figure  2-1  shows  when  each  activity  should 
occur.  It  is  generic,  and  is  not  reflecti\'e  of  actual  effort 
required  for  each  task.  Each  program  office  should  tailor  this 
schedule  of  activities  to  meet  the  needs  and  constraints  of  its 
specific  acquisition. 

Determining  Note  in  Figure  2-1  the  sigruficant  amount  of  time  allowed  for 

SCE  Use  determining  how  to  use  SCE.  Acquisition  organizations 

should  use  this  time  to  cor\sider  the  varying  methods  of 
implementation  and  the  individual  techniques  and 
procedures  which  may  be  used.  The  range  of  time  for  each 
activity  will  vary  with  the  experience  of  the  acquisition  team 
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regarding  SCE.  Those  orgartizations  familiar  with  SCE  use 
will  spend  significantly  less  time  on  the  intervening  steps 
once  the  decision  to  use  SCE  has  been  made. 


The  acquisition  organization  must  decide  how  to  factor  the 
SCE  findings  into  the  acquisition.  Part  B  of  this  doaiment 
explores  these  issues  and  others  a  program  office  must 
cortsider  to  use  SCE  in  a  source  selection.  Experience  has 
demonstrated  the  need  for  preparing  as  early  as  possible, 
particularly  in  large,  understaffed  acquisition  offices. 
Additional  preparation  time  may  need  to  be  factored  into  the 
schedule  in  anticipation  of  gaining  approval  to  proceed  with 
the  SCE.  For  example,  contracts  for  small  or  small, 
disadvantaged  businesses  require  extra  preparation  and 
approvals  outside  the  normal  chain  of  command. 


Selecting  the  Selecting  qualified,  experienced  software  acquisition 

SCE  Team  personnel  to  serve  as  SCE  team  members  is  one  of  the  most 

difficult  and  important  tasks  a  program  or  software  manager 
may  do  in  the  course  of  implementing  SCE  in  an  acquisition. 
The  key  considerations  for  assembling  an  SCE  team  are 

•  Individual  experience. 
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Individual 

Experience 


Team  Skills 
Experience 


•  Team  skills  and  experience. 

•  Individual  interpersonal  skills. 

•  Team  leadership  skills,  team  building  activities,  and  team 
staffing. 

Individual  experience  is  an  important  factor  in  choosing  team 
members.  SCE  team  members  should  be  selected  firom  the 
most  experienced  people  in  the  program  office,  qualified 
personnel  from  government  field  activities  (laboratories), 
qualified  engineering  support  contractors  (e.g.MITRE, 
Aerospace,  IDA),  and  people  firom  other  program  offices  that 
can  be  used  on  a  temporary  basis.  The  most  successful  teams 
will  be  those  that  average  at  least  seven  years  of  software 
and/or  software  acquisition  experience  across  the  team.  At 
least  one  member  of  the  team  should  have  source  selection 
experience.  This  is  important  because  what  can  and  cannot  be 
done  during  a  source  selection  is  different  from  what  is 
permissible  after  award.  An  acceptable  spread  of  experience 
levels  in  a  team  which  have  been  found  to  be  successful  is 

•  At  least  one  or  two  senior  personnel  (more  than  seven 
years  appropriate  experience). 

•  Two  or  three  mid-level  persoimel  (five  to  seven  years  ap¬ 
propriate  experience). 

•  One  j\mior  person  (two  to  four  years  appropriate  experi¬ 
ence)  Note:  This  is  a  recommended  maximum.  Junior  per¬ 
sonnel  typically  will  not  have  the  backgroimd  to  under¬ 
stand  certain  aspects  of  software  processes  they  will  ob¬ 
serve. 

The  background  of  SCE  team  members  should  strike  a 
balance  between  software  technical,  software  mariagement, 
and  software  acquisition  experience.  They  should,  as  a 
minimum,  share  a  mix  of  knowledge  and  experience  in  the 
following  software  engineering  disciplines: 

•  Acquisition  policies  and  procedures  (particularly  source 

lection  procedures). 
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Interpersonal  Skills 


Team  Leadership 
Skills 


•  Project  management  and  planning. 

•  Software  configuration  management. 

•  Software  design,  development  and  methodologies. 

•  Software  quality  assurance. 

•  Systems  engineering. 

•  Technical  requirements  of  the  contract. 

•  Testing. 

•  Application  domain,  e.g.  Avionics,  Missiles,  C3I,  databas¬ 
es. 

In  general,  expertise  will  be  necessary  from  at  least  one  team 
member  in  each  of  the  key  process  areas  to  be  investigated. 
Certain  areas  may  be  stressed  over  others  depending  on  the 
acquisition. 

SCE  team  members  must  be  "team  players."  Conducting 
SCEs  requires  professional  judgement  and  team  consensus; 
attributes  which  are  necessary  to  work  effectively  in  a  SCE 
team  are  patience  and  perseverance.  Past  experience  has 
demonstrated  that  if  team  manU?ers  lack  interpersonal 
skilb — essential  to  fostering  good,  open  conununications 
between  team  members  and  the  contractors — the  team  is  less 
effective,  less  credibile,  and  less  motivated  to  fulfill  the  SCE 
objectives.  The  ability  to  communicate  with  the  contractor 
and  other  team  members  is  the  essence  of  SCE  teamwork. 

Experience  shows  that  an  effective  team  leader  is  critical  to  the 
operation  of  the  team.  The  team  leader 

•  Ensures  that  the  team  meets  its  schedule  and  objectives, 
encourages  teamwork  and  consensus  building. 

•  Is  the  SCE  team  focal  point  for  both  the  acquisition  office 
and  the  contractors  (plarming,  scheduling,  conununicat- 
ing). 
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Team  Building 
Activities 


•  Should  have  enough  basic  leadership  skills  to  ensure  that 
the  team  functions  effectively. 

The  team  leader  should  be  the  one  most  qualified,  based  on 
knowledge,  experience,  and  amount  of  direct  SCE  site  visit 
experience.  Occasionally,  teams  can  break  down  when  an 
inappropriate  team  leader  is  appointed.  Program  office 
management  should  prevent  this  from  happening;  i.e.,  a 
lieutenant  should  not  be  put  in  charge  of  a  team  just  because 
he  or  she  wears  a  uniform.  Previous  SCE  experience  should  be 
a  criterion  for  assignment  as  a  SCE  team  leader.  New  SCE 
team  leaders  should  have  participated  on  at  least  two  SCEs  as 
a  team  member  prior  to  assuming  a  leadership  role.  This 
experience  of  participating  on  SCEs  will  prepare  the  new 
leader  to  understand  the  SCE  team  process,  team  dynamics, 
and  contractor  sensitivities. 

An  essential  aspect  of  preparing  a  team  to  conduct  SCE  site 
visits  is  performing  team  building  activities  prior  to  going  on 
site.  Assume  the  SCE  team  has  never  worked  together.  An 
activity  that  would  help  bring  the  individual  members 
together  as  a  team  could  be  an  exercise  whereby  a  simple  task 
is  assigned  and  discussed.  For  example,  each  team  member 
would  interview  the  p>erson  to  his  or  her  right  at  a  table  or  in 
a  room.  The  task  of  the  interviewer  is  to  learn  the  person's 
background,  interests,  and  area  of  expertise.  Each  team 
member  would  then  introduce  and  briefly  state  the  results  of 
their  interview.  The  team  could  then  identify  its  relative 
background  expertise  areas  to  the  evaluation  task  they  are 
being  asked  to  perform.  For  reference,  the  bibliography  in 
Appendix  C  of  this  guide  contains  three  references  [Bucholz 
87],  [Denton  89],  [Kelly  91],  that  contain  more  on  teams  and 
team  building  activities.  These  exercises  will  help  determine 
how  the  team  members  work  together.  Often,  many  months 
may  pass  after  teams  have  completed  SCE  team  training  and 
before  they  conduct  site  visits.  The  team  building  activities 
are  important  for  the  team  members  to  re-acquaint 
themselves  as  a  single  operating  entity  able  to  reach 
consensus  effectively.  There  may  be  times  when  trained 
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Team  Staffing 


Training  the  SCE 
Team 


individuals  are  brought  together  who  have  not  completed 
training  together.  In  this  scenario,  team  building  is  crucial, 
since  they  have  not  operated  as  a  team  before. 

The  success  of  the  SCE  team  hinges  on  its  ability  to  identify 
and  reach  consensus  on  the  strengths  and  weaknesses  of  a 
contractor.  A  SCE  team  is  neither  an  autocracy,  where  the 
leader  dictates  what  decisior\s  are  made,  nor  a  democracy, 
where  the  team  votes  and  the  majority  prevails.  Instead  the 
decisions  are  reached  by  team  consensus,  meaning  all 
members  agree  to  the  findings,  and  there  is  no  significant 
minority  dissent  on  issues.  If  consensus  on  an  issue  cannot  be 
reached,  then  there  is  no  finding  in  that  area.  This  is  where 
team  building  activities  will  pay  large  dividends. 

Staffing  the  team  is  another  issue  for  consideration.  ESC  has 
built  a  "software  center  of  excellence"  in  the  AVS 
organization.  ESC  has  also  endeavoured  to  build  a  core  of 
individuals  who  are  highly  skilled  in  conducting  SCE. 

System  Program  Offices  will  normally  draw  SCE  trained 
personnel  from  within  their  own  organization  and  their  "two 
letter"  pool.  If  this  pool  does  not  have  enough  individuals,  a 
request  to  the  AVS  organization  would  then  be  made  to  assist 
in  identifying  team  members.  In  this  marmer,  the  program 
office  can  take  advantage  of  key  components  mentioned 
above  under  individual  and  team  skills.  This  alternative 
makes  better  use  of  limited  software  skilled  resources  while 
ensuring  that  the  program  office  acquisition  expertise, 
knowledge  of  the  product  type  and  contractor  base,  and 
"domain"  knowledge  of  the  product  to  be  acquired  is  present 
on  the  team.  Furthermore,  the  core  resources  will  become 
valuable  assets  to  the  organization  as  they  gain  more 
experience  conducting  evaluations  for  multiple  acquisitions 
in  different  programs. 

Training,  preparation,  and  experience  separates  good  SCE 
teams  from  poor  ones.  There  are  three  levels  of  training 
needed  before  an  individual  should  be  considered  fully 
qualified  to  conduct  SCEs: 

•  Preparation 
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•  Coursework 

•  Experience 

Preparation 

The  preparatory  stage  consists  of  on-the-job  experience,  prior 
training,  and  professional  reading.  It  is  recommended  that 

SCE  team  members  take  courses  in  team  work,  assertiveness 

training,  and  total  quality  management.  Watts  Humphrey's 
book.  Managing  the  Software  Process  [Humphrey  89)  and  the 
latest  release  of  the  CMM  [Paulk  91a]  are  two  essential 
readings  that  are  provided  to  participants  of  the  team  training 
course.  Participation  in  the  one  day  SCE  Overview  Seminar  is 
also  benehdal  background  to  prepare  team  members. 

Coursework 

SCE  team  training  consists  of  a  multi-day,  case-study- 
oriented  course  that  all  SCE  team  members  must  successfully 
complete.  This  course  is  intended  for  teams  of  four  to  six 
experienced  personnel  who  have  been  selected  to  conduct  the 

SCE  site  visits.  It  provides  the  knowledge  and  reinforces  the 
skills  required  to  effectively  conduct  SCEs,  and  helps  the 
group  develop  into  a  cohesive  team.  A  sampling  of  course 
content  follows: 

•  Team  building  exercises. 

•  Preparing  for  the  site  visit. 

•  Conducting  interviews. 

•  Substantiating  key  practices. 

•  Reviewing  documents. 

•  Developing  and  presenting  findings. 

SCE  teams  need  effective  communicators  willing  to  take  on 
different  roles  (e.g.  facilitator,  recorder,  interviewer, 
timekeeper),  as  demanded  by  the  dynamics  of  the  team  and 
constraints  of  the  acquisition.  The  SCE  team  needs  to  know 
how  to  evaluate  the  contractor  in  relation  to  the  Capability 

Maturity  Model,  so  a  working  understanding  of  the  CMM  is 
required.  Teams  are  taught  that  processes  implemented  are  to 
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Experience 


a  large  degree  dependent  on  several  non-process  variables.  It 
takes  experience  to  understand  these  relationships  and 
exercise  professional  judgement,  which  is  why  the  team 
characteristics  and  profile  are  crucial  in  addition  to  the 
coursework.  Roles  taken  on  by  team  members  to  accomplish 
the  site  visit  include  the  following; 

•  Team  Leader:  manages  process,  keeps  to  agenda,  guaran¬ 
tees  deadlines  and  deliverables. 

•  Facilitator;  sustains  team  spirit,  moves  team  toward  con¬ 
sensus,  and  encourages  participation. 

•  Recorder;  captures  information,  does  not  editorialize,  and 
lists  documents  to  be  reviewed. 

•  Participant:  assists  other  team  members  and  carries  out  as¬ 
signed  tasks. 

Every  graduate  of  the  SCE  training  program  should  be  a 
member  rather  than  a  leader  of  an  SCE  team  for  at  least  two 
SCEs.  Junior-  and  mid-level  personnel  should  take  part  in  at 
least  three  SCEs  before  being  considered  for  the  team 
leadership  role.  Resource  demands  and  time  constraints, 
however,  may  prevent  this  from  happening.  When  working 
under  such  constraints,  it  is  recommended  that  the  program 
office  send  the  team  to  practice  an  SCE  on  a  government  office 
before  beginning  the  source  selection.  One  goverrunent 
acquisition  team  practiced  doing  SCEs  on  at  least  three 
occasions  to  ir\sure  personnel  were  highly  trained  for  the 
source  selection. 

The  common  denominators  in  discussions  with  individuals 
returning  from  their  first  SCE  is  their  desire  for  more  team 
training,  preparation,  and  time  to  conduct  the  interviews. 
SCE's  activities  are  not  radically  different  from  those  done  in 
the  program  office  on  a  day  to  day  basis.  Taken  together, 
however,  they  are  group  activities  requiring  significant 
practice  and  preparation.  Practicing  as  a  group  will  reveal 
individuals'  strengths  and  weaknesses,  depth  of  required 
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preparation,  and  how  to  manage  the  SCE  process  to  capitalize 
on  the  team's  strengths  so  that  more  effective  and  timely  SCEs 
are  conducted. 


Providing 
Directions  to  the 
Potential 
Offerors 

Acquisition 

Documents 


SCE  in  source  selection  provides  the  program  office  with 
several  opportunities  to  interact  with  the  potential  offeror(s). 
The  program  office  will  need  to  provide  directioits  in  the 
Instructions  For  Proposal  Preparation  (IFPP)  explaining  how 
to  complete  the  preparatory  materials  that  form  the  initial 
baseline  of  the  site  visit.  They  will  also  have  to  present  the  SCE 
process  and  describe  how  the  findings  are  factored  into  the 
acquisition  at  either  a  Pre-proposal  Conference  or,  in  the 
acquisition  documents  or  both.  Part  B  of  this  document  will 
discuss  in  detail  the  areas  requiring  govenunent  direction  and 
interaction  with  the  contractor. 

For  this  discussion  the  use  of  SCE  will  affect  the  following 
acquisition-related  documents.  Documents  with  an  asterisk 
{*)  after  them  provide  direction  and  information  to  the 
contractor  community  in  an  acquisition,  while  the  others  are 
government  internal  documents. 

•  Commerce  Business  Daily  Announcement* 

•  Source  Selection  Plan  (SSP) 

•  Source  Selection  Evaluation  Guide  (SSEG) 

•  Pre-Proposal  Conference  * 

•  Request  for  Proposal  (RFP)*  -  Sections  H,  M,  L,  (IFPP) 

•  Possibly — Statement  of  Work  (SOW)  or  Award  Fee  Plan* 

•  Debriefings  of  Unsuccessfixl  Offerors* 
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Contractor  Project 
Profiles  and  Maturity 
Questionnaire 
Responses 


Requests  for  Other 
Information 


The  acquisition  organization  will  request  the  offerors  to 
prepare  and  submit  profiles  and  Maturity  Questionnaire 
responses  for  each  of  six  to  eight  projects  that  are 
representative  of  the  software  development  work  at  the  site(s) 
that  is  being  proposed.  This  is  discussed  further  in  Part  B  of 
this  guide.  For  the  remainder  of  this  chapter  discussion,  only 
the  highlights  of  preparation,  conducting  the  site  visit  and  the 
final  reports  will  be  covered. 

The  contractor's  organization  charts  specific  to  the 
organizational  site  and  the  projects  to  be  submitted  for 
corwideration  by  an  SCE  team  are  particularly  helpful  in 
building  a  preliminary  plan  of  who  is  to  be  interviewed  to 
discuss  certain  issues.  Request  after  competitive  range 
determination  and  before  site  visits  only  that  documentation 
which  the  SCE  team  has  time  scheduled  to  review  and  factor 
into  its  plaiming  activities.  Providing  documentation  is  a 
burdensome  task  for  the  contractors  and  reviewing  it  is  a 
time-consuming  activity  for  the  SCE  team.  Select  only  what  is 
essential  based  on  analysis  of  contractors'  responses  and  only 
if  time  is  available.  Documents  that  will  reveal  processes  at 
work  should  be  selected  over  those  that  are  techiucal  in  nature 
or  discuss  plans.  (Plai\s  often  are  not  implemented  or  updated 
and  are  good  only  as  a  point  of  departure.) 


Preparing  and 
Conducting  the 
Site  Visit 


Preparing  for  the 
Site  Visit 


The  following  discussion  is  a  brief  overview  of  the  essential 
steps  required  to  execute  the  SCE  on-site  period.  It  is  included 
here  for  continuity  of  the  Part  A  Discussion.  Detailed 
discussion  are  available  in  a  separate  SEI  Document,  the 
Software  Capability  Evlauation  Version  1.5  Method  Description 
[SCE  93]. 

•  Developing  the  acquisition  product  profile  and  Target 
Process  Capability.  The  Target  Process  Capability  is  the 
explicit  identification  of  the  software  process  scope  to  be 
evaluated.  SCE  uses  the  KPAs  of  the  CMM  to  "target"  the 
areas  of  software  process  capability  investigation.  The  rec- 
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Conducting  Site 
Visits  (For  Each 
Contractor) 


ommended  minimum  set  of  KPAs  for  evaluation  are  the 
KPAs  associated  with  Repeatable  and  Defined  mahurity 
levels. 

•  Selecting  contractor  projects. 

•  Identifying  Critical  Subprocesses  for  all  contractors. 

•  Analyzing  the  contractor's  project  profiles  and  Maturity 
Questionnaire  responses  with  respect  to  the  product  pro¬ 
file  and  the  TPC.  Note:  The  Maturity  Questionnaire  is  used 
only  as  an  input  in  conjimction  with  the  analysis  of  project 
projfiles  with  respect  to  the  product  profile  and  the  Target 
Process  Capability.  The  Maturity  <3uestiormaire  is  not 
scored  by  the  SCE  team  and  does  not  have  direct  impact 
on  the  SCE  findings. 

•  Determining  key  issues  for  individual  contractors. 

•  Developing  initial  exploratory  interview  questions. 

•  Developing  an  agenda  and  schedule  for  initial  exploratory 
interviews  and  document  reviews 

•  Notifying  individual  contractor  focal  points  of  SCE  team 
logistic  requirements. 

•  Presenting  the  arrival  brief  to  contractor  management. 

•  Analyzing  organizational  and  project  documentation. 

•  Reviewing  agenda  and  schedule. 

•  Conducting  exploratory  interviews. 

•  Requesting  additional  documentation. 

•  Validating  interview  responses. 

•  Drafting  preliminary  findings. 

•  Validating  preliminary  findings. 

•  Conducting  consolidation  interviews. 
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•  Validating  improvement  activities. 

•  Collecting  final  data. 

•  Developing  final  findings. 

•  Concluding  meeting/  (as  prescribed  by  Procuring  Con¬ 
tracting  Officer  (PCO)). 

Development  of  Using  its  collective  professional  judgement  and  a  consensus 

Findings  decision  making  process,  the  SCE  team  puts  together  its 

findings  from  individual  projects  to  aeate  a  set  of  overall, 
organization-level  findings.  These  findings  are  the 
embodiment  of  all  the  interview  and  documentation  notes 
developed  before  and  during  the  on-site  review.  Detailed 
findings  should  be  prepared  for  each  JCPA  investigated. 

Findings  should  be  specific  to  the  point  where  they  identify 
the  cause  for  a  strength  or  weakness,  but  not  so  specific  that 
the  finding  places  the  team  in  a  comer  by  failing  to  consider 
exceptions  tnat  may  exist  within  the  organization.  SCE  teams 
must  remember  they  are  evaluating  a  subset  of  the  total 
projects  ongoing  at  a  site  as  a  proxy  for  predicting  the 
organization's  capability  to  do  a  specific  project,  and 
exceptions  may  exist  because  of  this  process.  The  team  should 
be  prepared  to  substantiate  the  strengths  and  weaknesses 
without  attributing  the  information  to  specific  individuals  or 
projects.  Individual  confidentiality  is  a  vital  component  of  a 
good  site  visit.  The  team  should  create  and  maintain  a 
detailed  record  of  the  site  visit  which  the  contracting  officer 
can  include  as  part  of  the  documentation  making  up  the 
permanent  contract  file. 
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Part  B  Using  SCE  in  a  Source  Selection 

Introduction  The  next  three  chapters  explore  in  depth  the  role  of  SCE 

throughout  the  source  selection  process.  First  a  high  level  look 
is  provided,  then  some  key  issues  w^hich  should  be  considered 
when  using  SCE  are  explored.  Part  B  will  provide  the  reader 
a  realistic  understanding  of  the  extensive  planning  required 
to  implement  SCE  in  a  source  selection.  SCE  is  more  than  a 
series  of  site  visits  followed  by  an  outbrief  to  the  Source 
Selection  Advisory  Council  (SSAC),  Source  Selection 
Evaluation  Board  (SSEB)  or  equivalent  organization  under 
streamlined  source  selection  procedures. 

Figure  B-1  provides  a  high  level  flow  chart  of  the  normal 
source  selection  activities  that  will  be  affected  by  the 
incorporation  of  SCE  into  the  source  selection  process.  The 
chart  simplifies  somewhat  the  multitude  of  interactions  that 
go  on  during  the  planning  and  execution  of  SCE.  The  first  step 
in  the  process  is  whether  or  not  SCE  should  be  used  in  the 
source  selection  process. 

Chapter  3  places  SCE  in  the  context  of  a  typical  source 
selection  by  discussing  issues  which  should  be  considered. 
These  issues  include  criteria,  costs  and  benefits  of  using  SCE, 
both  for  the  acquisition  organization  and  for  the  offerors. 
Chapter  3  also  looks  at  the  various  persoruiel  involved  in  the 
SCE  process. 
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Figure  B-1  Source  Selection  Activities  Affected  By  SCE 

Once  the  organization  has  decided  to  use  SCE  in  an 
acquisition,  the  proper  role  for  SCE  in  the  source  selection 
criteria  must  be  assessed.  This  decision  point  is  labeled  with  a 
"1"  in  Figure  B-1.  Should  SCE  be  factored  in  as  a  Performance 
Risk  Assessment  or  Evaluation  Criterion  or  both?  Chapter  4 
discusses  some  altertiate  methods  of  using  SCE  findings  in  the 
source  selection  decision.  It  also  addresses  the  implications  of 
using  SCE  on  the  source  selection  schedule. 

The  boxes  labeled  2  through  5  address  documentation 
typically  required  for  a  source  selection,  whether  SCE  is  used 
or  not:  Source  Selection  Plan  (SSP),  Pre-proposal  Conference 
presentation.  Evaluation  Standard,  and  the  Request  For 
Proposal  (RFP).  Each  piece  of  documentation  must  address 
SCE  to  a  certain  degree.  Chapter  5  explains  how  to  develop 
these  documents  to  accommodate  SCE. 
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Like  all  schedules,  timetables,  or  flow  charts  found  in  this 
guide.  Figure  B-1  is  only  a  representation  giving  the  user  of 
SCE  insight  into  what  is  required  to  implement  SCE 
effectively.  Variations  to  the  schedules  and  timetables  should 
be  expected.  For  example,  SCE  training  could  easily  be  carried 
out  before  the  Pre-proposal  Conference  and/or  the  writing  of 
the  Evaluation  Standard  in  order  to  accommodate  the  unique 
demands  of  the  acquisition. 
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Chapter  3  Deciding  to  Use  SCE 


Criteria  for 
Using  SCE  in  a 
Source 
Selection 


This  section  introduces  the  issues  that  must  be  considered 
when  contemplating  use  of  SCE  in  source  selection.  General 
familiarity  with  the  government's  source  selection  process  is 
assumed  for  purposes  of  focusing  on  each  player's 
relationship  to  SCE.  Appendix  A;  SCE  Participants  in  Source 
Selection  provides  a  brief  discvission  of  each  primary  Source 
Selection  participant  that  is  affected  or  can  affect  the  use  of 
SCE. 

Clearly,  SCE  should  not  be  used  for  every  software 
acquisition  in  the  government.  Both  the  costs  and  benefits  of 
using  SCE  as  well  as  the  specific  nature  of  the  acquisition 
should  be  considered  when  making  this  decision.  These  costs 
and  benefits  may  indicate  that  other  approaches  are  necessary 
for  very  small  acquisitions  involving  software.  This  section 
discusses  criteria  related  to  the  nature  of  an  acquisition. 

There  is  no  minimum  number  of  lines  of  code  or  measure  of 
software  intensity  dictating  the  use  of  SCE  in  an  acquisition. 
Several  considerations  must  be  weighed  by  the  program 
manager  when  making  the  decision.  Each  of  the  following 
factors  are  important  considerations,  but  the  program 
manager  or  person  responsible  for  determining  SCE  usage  for 
an  acquisition  must  weigh  these  factors  in  accordance  with 
their  organization's  method  of  procuring  systems.  These  are 
general  guidelines  which  must  be  refined  to  meet  the  context 
of  the  organization: 

•  Criticality  of  an  acquisition  or  the  software  component. 

•  Total  dollar  value  of  the  acquisition  or  software  compo¬ 
nent. 

•  Management  control  priority. 

•  Unprecedented  system  mission  needs. 
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•  Acquisition  life  cycle  phase. 

•  Length  of  acquisition  time  period. 

•  Software  size,  the  number  of  Computer  Software  Compo¬ 
nents  (CSCs),  and  the  prime  contractor  -  subcontractor  re¬ 
lationship. 

Figure  3-1  illustrates  the  relationship  of  each  of  these  factors 
as  a  general  guideline  for  determining  appropriateness  of  SCE 
usage.  Each  box  should  be  read  independently,  and  then 
combined  with  other  factors,  to  make  an  overall  judgement  on 
SCE  applicability. 
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Component 


Total  Dollar  Value  of 
the  Acquisition  or 
Software 
Component 


Management 
Control  Priority 


CMU/SEI-93-TR-18 


The  criticality  of  an  acquisition  may  necessitate  SCE  use.  The 
SEI  recommends  that  any  DoD-defined  major  program  use 
SCE  as  an  integral  part  of  their  strategy  for  producing  the 
highest  quality  end  product  and  motivating  DoD  contractors 
to  focus  on  software  process  improvement  as  a  means  to  effect 
this  goal.  All  Mission  Critical  Computer  Resource  (MCCR) 
systems,  regardless  of  total  dollar  amount,  software  size,  or 
DoD  priority  ranking,  should  strongly  consider  SCE  usage. 
MCCR,  and  software  in  general,  are  critical  components  of 
modem  weapon  systems.  The  success  of  the  system  is  largely 
dependent  upon  software  precisely  performing  its  intended 
function.  An  example  of  a  small,  but  highly  critical  piece  of 
software  warranting  the  use  of  SCE  in  an  acquisition  would 
be  software  needed  to  control  the  hardware  for  an  access 
control  system  for  nuclear  weapons  or  other  munitions. 

SCE  should  be  used,  even  in  non-MCCR  systems,  when  the 
total  value  of  an  acquisition  software  component  exceeds  $10 
million.  Dollar  value  is  important  because  of  the  investment 
in  resources  and  time  necessary  to  implement  SCE  effectively 
and  because  acquisition  personnel  need  a  common  frame  of 
reference  against  which  to  compare  usage.  On  any  acquisition 
in  which  total  cost  is  greater  than  $25  million,  which  includes 
a  software  component  of  any  amount,  SCE  use  should  be 
strongly  considered. 

This  threshold  should  not  be  construed  as  an  absolute.  Where 
the  software  component  of  a  program  itself  exceeds  $10 
million  or  is  greater  than  30%  of  the  total  program  cost,  SCE 
should  be  used.  This  criterion  may  often  take  precedence  over 
the  $25  million  threshold  described  above.  On  the  other  hand, 
some  acquisitions  in  the  $10  to  $25  million  range  may  not 
warrant  the  use  of  SCE  because  of  program-unique 
circumstances.  Perhaps  the  software  component  is  not 
mission  critical  and  is  less  than  10%  of  the  total  dollar  value. 
There  is  no  firm  guideline. 

When  management  control  is  a  high-priority  concern,  SCE 
use  should  be  considered.  An  environment  under  effective 
management  control  wiU  be  more  able  to  produce  data  that  is 
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useful  for  lessons  learned  which  can  be  incorporated  into  the 
overall  systems  development  process.  These  lessons  help  the 
DoD  avoid  "re-inventing  the  wheel."  Successful  management 
control  also  facilitates  effective  implementation  of  modern 
methodologies,  tools,  and  techniques.  A  controlled 
environment  is  essential  to  managing  contractor  processes — 
processes  for  maintaining  the  development  environment, 
bringing  new  people  and  technology  into  the  environment, 
identifying  problems  early  in  the  contract,  and  managing 
requirements  changes.  A  controlled  environment  enables 
improved  risk  assessment  and  abatement,  and  the  capability 
to  measure  management  control  of  an  organization's  software 
development  process  is  a  primary  benefit  of  using  the  SCE 
Method.  Each  program  manager  must  exercise  judgement  in 
this  area  to  determine  if  management  control  is  critical 
enough  to  warrant  SCE  usage. 

SCE  should  be  used  when  the  contractor  is  likely  to  develop 
software  implementations  which  are  imprecedented.  When 
the  mission  of  the  system  being  procured,  especially  the  role 
played  by  the  software  component,  is  known  and  has  been 
defined  by  the  end  user,  and  portions  of  the  system  wiU  be 
unprecedented,  SCE  use  will  help  identify  program  risks 
associated  with  the  capability  of  contractors  to  succeed  in 
producing  quality  software  in  an  unprecedented 
environment.  SCE  site  visits  yield  information  about  an 
organization's  ability  to  manage  risks  inherent  in 
unprecedented  software  development,  as  well  as  their  ability 
to  manage  tasks  which  are  new,  but  are  sirrular  to  ones  they 
have  successfully  completed  previously.  This  SCE 
information  can  then  be  used  to  support  color  codes  assigned 
for  Technical  Area /Items  to  identify  software  impacts  on 
system  performance. 

Unprecedented  systems  (i.e.,  those  solving  new  or  unique 
problems)  pose  special  problems  for  software  development 
organizations.  An  SCE  of  each  offeror  would  identify  whether 
the  requisite  controls  are  in  place  on  the  contractors'  existing 
programs  and  whether  they  will  be  easily  transferred  to  the 
new,  unprecedented  system. 
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The  lifecycle  phase  of  an  acquisition  is  an  important  factor  in 
determining  SCE  usage.  The  SCE  Method  and  CMM  were 
originally  developed  in  response  to  DoD's  and  industry's 
recognized  problems  in  managing  the  development  of 
increasingly  complex  mission  critical,  software  intensive 
products  in  the  real-time,  embedded,  aerospace  domain. 
Given  this  background,  SCE  fits  in  any  Engineering 
Manufacturing  Development  (EMD)  program  within  this 
domain,  since  EMD  is  the  typical  phase  associated  with  major 
new  software  development.  The  SEI  recommends  that  any 
EMD  program  consider  SCE  use,  in  accordance  with  the  other 
factors  listed  here.  However,  SCE  use  is  not  in  any  way 
limited  to  the  EMD  phase  alone.  In  fact,  since  SCE  became 
publicly  available  in  1987  in  a  preliminary,  pilot  version,  it  has 
been  successfully  (as  reported  by  acquisition  orgaiuzations) 
used  in  several  other  phases.  Outside  EMD,  there  are  more 
context-sensitive  factors  to  consider  whether  SCE  use  is 
appropriate. 

The  SCE  Method  should  be  considered  in  any  procurement 
where  software  is  a  major  component  and  the  program 
duration  period  is  expected  to  be  greater  than  eighteen 
months.  This  timeframe  is  recommended  because  of  the 
amount  of  resources  necessary  to  apply  SCE  effectively,  and 
becavtse  the  typical  process  improvement  program 
implemented  by  a  contractor  in  response  to  the  source 
selection  use  of  SCE  generally  requires  at  least  18-24  months 
to  attain  and  sustain  moves  to  the  next  higher  process 
maturity  level.  Thus,  more  software  development  time  is 
necessary  to  see  improved  results  directly  on  the  contract. 

SCE  should  also  be  cor\sidered  when  the  program  office 
expects  significant  block  upgrades,  modifications,  or  follow- 
on  programs  to  occur,  and  the  original  contractor  is  expected 
to  be  a  primary  offeror  or  likely  performer  of  the  new  work. 
Often,  the  processes  put  in  place  by  the  contractor  at  the  start 
of  a  software  development  will  be  frozen,  meaning  that 
process  changes  will  be  limited  during  that  development 
period.  Software  upgrades  or  major  modifications  to  existing 
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systems  are  good  times  to  unfreeze  the  current  software 
development  process  and  install  new,  improved  processes, 
methods,  and  technology.  Therefore,  using  SCE  during  the 
initial  software  development  and  the  subsequent 
improvement  programs  will  enable  any  improved  processes 
to  be  implemented  on  the  follow-on  developments. 

SCE  use  may  still  be  appropriate  even  if  neither  of  these 
criteria  are  met  if  the  Program  Executive  Officer  (PEO) 
center/commander  or  activity  committee  is  attempting  to 
motivate  and  gain  improvements  in  a  particular  domain  area, 
such  as  avionics  systems.  These  PEO  decisions  may  entail 
long-range  considerations  which  go  beyond  the  current 
contract  award,  and  thus  SCE  use  may  be  appropriate  to  meet 
other  government  objectives. 

The  amoimt  of  software  to  be  developed  will  bear  a  direct 
relatiortship  to  the  number  of  CSCs  required  to  effectively 
partition  the  software  system  into  manageable  chunks  which 
can  be  built  by  development  teams,  and  also  on  the  likelihood 
of  a  prime  contractor  performing  integration  of  software 
produced  by  several  subcontractors.  When  the  estimated  size 
of  the  software  component  exceeds  100  thousand  source  lines 
of  code  (KSLCXT),  SCE  should  be  used.  At  this  threshold,  the 
complexity  of  the  software  development  will  likely  cause  the 
ability  to  manage  a  large  number  of  CSCs  and  subcontractors 
to  be  a  significant  concern  of  the  program  manager.  Scaling 
up  small  software  engineering  teams  to  meet  the  challenges  of 
a  large  development  creates  additional  and  greater  pressures 
on  effective  software  development  processes  to  facilitate  cost, 
schedule,  and  performance  parameters  of  the  program  to  be 
successfully  met.  There  are  several  considerations  that  should 
be  weighed  by  the  program  manager: 

•  Software  size  between  25  and  200  thousand  source  lines  of 
instructions. 


•  Minimum  development  team  of  20  to  100  people  in  under 
a  year  with  several  years  of  support  and  enhancements. 
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•  Software  embedded  on  multiple  platforms  in  different 
languages  for  a  real-time  application. 

•  Highly  specialized  technologies:  for  example,  radar  signal 
processing  on  a  unique  programmable  signal  processor  or 
image  processing  on  a  customized  parallel  processor. 

•  Software  pieces  to  be  subcontracted  to  geographically  dis¬ 
tant  locations. 

These  examples  highlight  different  managerial/technical 
capabilities  a  contractor  must  possess  depending  on  the  type, 
complexity,  and  size  of  the  software  and  the  nature  of  the 
delivery  schedule. 

A  strong  understanding  of  acquisition-specific  circumstances, 
rather  than  hard  criteria,  is  necessary  to  determine  whether 
SCE  use  is  appropriate.  In  general,  for  all  acquisitions  with  a 
software  component,  the  government  should  seek  to  do 
business  with  contractors  who  understand  and  effectively 
implement  modem  software  development  practices,  and  also 
with  those  contractors  taking  actions  to  improve  these 
practices.  SCE  is  a  tool  which  can  augment  other  government 
techniques  for  discerning  ditferences  in  the  capabilities  of 
offerors,  and  is  essential  for  evaluating  contractor  software 
process  capability  and  process  improvement  plans. 

The  SEI  has  been  piloting  the  SCE  Method  with  organizations 
in  the  Army,  Air  Force,  and  Navy  since  its  inception  in  1987. 
Several  orgaiuzations  have  begxm  to  establish  criteria  for  SCE 
use  which  reflect  the  individual  needs  of  these  organizations, 
and  supplement  the  information  contained  in  this  section  of 
the  guide.  One  major  command  has  drafted  a  policy  requiring 
SCE  use  on  all  MCCR  programs  exceeding  $10  million. 
Another  military  division  is  taking  the  approach  of  requiring 
SCE  use  on  procurements  which  include  software  size 
estimates  greater  than  50  KSLOC.  These  examples  underscore 
the  importance  of  refining  SCE  usage  criteria  to  best  reflect 
the  acquisition  practices  implemented  at  a  particular 
government  organizational  site.  Different  organizations  wiU 
have  different  business  bases,  contractor  communities. 
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product  types,  application  domains,  etc.,  all  of  which  affect 
site-specific  implementing  instructions  for  SCE.  No  standard 
DoD  policy  exists  as  of  the  printing  of  this  version  of  the 
gtiide. 

Pilot  use  of  the  SCE  Method  in  Army,  Navy,  and  Air  Force 
indicates  that  SCE  helps  the  acquiring  organization  in  many 
ways; 

•  Added  software  development  capability  realism  in  the 
source  selection  process. 

•  Increased  objectivity  in  information  collected  for  an  acqui¬ 
sition. 

•  Motivation  for  contractor  software  process  improvement 
actions. 

Software  Development  Capability  Realism:  One  benefit  SCE 
provides  over  other  contractor  software  development 
capability  evaluation  activities  done  during  the  typical  source 
selection  is  the  software  development  capability  realism 
introduced  into  the  proposal  review  and  contractor  analysis 
process.  The  information  SCE  collects  is  timely,  real  and  based 
on  current  projects  and  the  practices  actually  being 
implemented  by  offerors'  engineering  and  managerial 
persormel. 

For  moderate  to  large  software  development  efforts,  a 
currently  popular  means  of  evaluating  a  contractor's  software 
development  abilities  during  a  source  selection  is  the  review 
of  the  offeror's  Software  Development  Plan  (SDP). 
Comparing  the  SCE  findings  with  the  evaluation  of  the 
contractor's  proposal  and  SDP  will  clarify  for  the  program 
office  whether  the  proposed  approach  is  realistic  in  light  of 
the  offeror's  current  process  capability.  Based  on  this 
comparison,  the  program  office  can  better  evaluate  the  risks 
posed  by  each  offeror  amd  work  with  the  winning  contractor 
on  a  realistic  software  process  improvement  program. 
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Objectivity:  A  second  major  benefit  of  SCE  is  the  objectivity  it 
introduces  in  the  proposal  review  process.  The  SCE  Method 
helps  eitsure  an  objective  review  by  putting  a  trained 
evaluation  team  on  site  to  evaluate  the  offerors  activities  and 
compare  them  against  a  public  standard,  the  CMM.  In  the 
typical  source  selection,  evaluating  software  risk  is  a  difficult 
task  because  there  are  few  avenues  for  addressing  this  issue 
other  than  by  evaluating  what  is  in  the  proposal. 

With  the  goals  of  the  CMM  KPAs  as  a  basis,  contractor 
software  process  capability  can  be  reliably  measured  against 
a  common  standard.  This  permits  consistent,  repeatable,  and 
fair  evaluation  of  contractor  software  process  capability, 
adding  value  to  the  source  selection  process  by  making 
historically  ad  hoc  software  reviews  more  objective  and 
comparable  across  programs. 

Motivation  for  contractor  software  process  improvement 
actions:  In  order  to  remain  competitive  on  successive 
acquisitions,  contractors  must  improve  their  software 
development  processes.  In  contract  monitoring,  capability 
evaluation  can  be  used  to  assure  that  contractors'  software 
process  maturity  is  the  same  or  has  changed  relative  to  that 
measured  during  source  selection,  augmenting  an  action  plan 
for  improvement.  The  PRAG  can  evaluate  process 
improvement  based  on  past  performance  risk  assessments  of 
the  software  process. 

By  making  SCE  a  discriminator  in  conducting  acquisitions, 
government  program  offices  will  motivate  contractors  to 
focus  on  software  process  capability  as  a  means  of  retaining  or 
increasing  government  business.  Given  the  premise  that 
product  quality  will  follow  process  quality,  focusing  on 
software  process  improvements  resulting  in  increased 
process  maturity  will  increase  the  likelihood  of 

•  Accvurate  estimates. 

•  Decreased  variance  among  projects. 

•  Reduced  cost  arid  schedule  targets. 
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Although  there  is  no  definitive  study  validating  these  benefits 
quantitatively,  there  is  significant  anecdotal  evidence  from 
individual  goverrunent  and  industry  organizations  to  suggest 
these  benefits  are  real. 

A  focus  on  improving  software  process  capability  should  lead 
to  error  prevention,  earlier  detection  of  errors  in  the  lifecycle, 
and  an  ability  to  manage  requirements  changes  effectively. 
Improvement  in  processes  which  yield  earlier  detection  of 
errors  and  better  management  of  the  requirements  change 
process  should  save  the  government  money  over  the  lifecycle 
of  major  systems. 

Using  SCE  in  a  source  selection  requires  persormel  and 
financial  resources,  on  both  the  contractor  and  government 
sides.  The  resource  considerations  affecting  the 
implementation  of  SCE  are 

•  Persormel 

•  Time 

•  Financial 

•  Offeror  resource  requirements 

This  section  doses  with  a  discussion  of  some  suggestions  for 
minimizing  costs  while  maintaining  the  benefits  of  SCE  use. 
Figure  3-2  shows  the  estimated  government  labor,  in  person 
days,  required  to 

•  Implement  SCE  in  program  documentation. 

•  Train  SCE  team  members. 

•  Conduct  the  site  visits. 

•  Incorporate  the  SCE  findings  into  source  selection  results  / 
decisions. 

The  estimate  assumes  a  single  source  selection,  a  program 
office  having  no  prior  experience  with  SCE,  and  three  offerors 
within  the  competitive  range  who  must  be  evaluated.  For  a 
team  of  five  who  conduct  three  site  visits,  the  total  labor  is 
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approximately  115  person  days.  For  reference,  the  estimated 
labor  for  an  acquisition  involving  only  one  site  visit  is  65 
person  days  {Total  Effort  Fixed  Costs  39.75  person-days  plus 
Variable  Cost  Effort  of  25  person-days  for  sit^^  visit).  Certainly, 
there  are  economies  of  scale  and  there  are  many  non¬ 
recurring  costs,  such  as  team  training,  which  will  continue  to 
reduce  overall  government  labor  costs  as  trained  resources 
are  utilized  on  subsequent  acquisitions.  In  an  instance  where 
the  program  manager  and  SCE  team  have  been  trained  and 
have  used  the  method  previously,  the  estimated  labor  to 
implement  SCE  on  an  acquisition  (with  three  site  visits) 
declines  to  83.5  person  days.  (114.25,  less  SCE  information 
gathering  5.25,  less  RFP  preparation  1,  less  SCE  Training  25) 

This  analysis  leaves  it  up  to  the  individual  program  office  to 
determine  their  own  average  person  cost  per  day,  average 
travel  and  per  diem  costs,  and  subsequently  add  these  to  the 
cost  of  team  training  to  estimate  a  total  dollar  cost  for 
implementing  SCE. 
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Who  Does  It 

Effort  days  per 

Number  of 

Total  Effort 

Phase 

person 

People 

SCE  informa- 

PM.  PCO. 

0.25 

3 

0.75 

tion  gathering 

SCE  team  lead 

1.5 

3 

4.5 

RFP  Prepare- 

SCE  team 

1 

1 

1 

tion 

leader  or  PM 

1 

1 

1 

.5 

1 

0.5 

.5 

1 

0.5 

Fixed  Costs 

SCE  Training 

SCE  team 
members 

5 

5 

25 

SCE  Findings 

SCE  team 

1 

5 

5 

Preparation 

members 

Contractor 

PM.  PCO, 

0.5 

3 

1.5 

Debriefs 

SCE  team 
leader 

Subtotals 

11.25 

39.75 

Variable  Cost 

SCE  Site 

SCE  team 

5 

5 

75 

VisKs  3 

members 

Total  Person  Days  Effort 

114.75 

Figure  3-2  Estimated  SCE  Labor  For  One  Source  Selection 

Personnel  The  largest  constramt  on  the  government  is  the  labor  effort 

Constraints  expended  by  the  individuals  constituting  the  SCE  team.  This 

team  is  needed  for  one  full  work  week  for  every  SCE  site  visit 
that  is  performed.  In  addition,  several  person-days  are 
needed  to  prepare  for  each  site  visit  and  prepare  the  detailed 
report  or  set  of  findings  that  is  submitted  to  the  management 
body  (SSEB  or  SSET)  ordering  the  evaluation. 

In  addition  to  the  site  visit  requirements,  the  SCE  team  leader 
or  the  program  office's  software  focal  point  will  be  needed  on 
a  part-time  basis  prior  to  the  site  visits  to  incorporate 
appropriate  language  into  the  source  selection  materials  that 
are  affected  by  SCE,  assemble  a  SCE  team,  and  schedule 
training  for  any  untrained  team  members.  This  part-time  task 
will  be  minimal  once  the  respective  acquisition  organization 
has  put  in  place  support  materials  for  SCE,  including  this 
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guide.  After  the  site  visits,  the  SCE  team  Leader  will  likely  be 
needed  to  advise  the  SSEB  and  SSET,  and  perform  outbriefs  to 
the  winning  and  unsuccessful  offerors  as  directed  by  his  or 
her  PCO.  Figure  3-3  shows  approximately  the  distribution  of 
human  resources  over  time. 


Figure  3-3  SCE  Manloading  Over  Time 

Time  Constraints  The  SCE  team  is  needed  for  at  least  one  and  a  half  weeks  for 

every  offeror  requiring  a  site  visit.  This  includes 

•  Preparation;  1-2  days. 

•  Travel  time;  1  day. 

•  Site  visit;  3  days  (includes  caucus  and  findings  prepara¬ 
tion). 

•  Time  off  between  site  visits:  1  day. 
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Time  off  is  important  because  site  visits  are  very  intense 
activities.  These  resource  loading  estimates  are  included  in 
Figure  3-3,  above. 

Another  time  constraint  is  imposed  by  the  typical  source 
selection  schedule.  Site  visits  cannot  begin  until  after  tl^ 
initial  proposal  evaluation  and  only  on  those  offerors 
remaining  in  the  competitive  range.  This  typically  allows  a 
one  to  two  month  window  for  the  conduct  of  the  on  site  phase 
of  the  SCE.  A  program  manager  does  not  know  the  number  of 
offerors  until  proposals  are  received.  This  mear»s  that  the 
program  manager  will  have  to  estimate  how  much  time  is 
needed  to  complete  all  the  SCEs  based  on  the  estimated 
number  of  offerors. 

Figure  3-4  presents  an  actual  cost  summary,  $61,400  (not 
including  labor  cost)  from  a  single  acquisition  that  conducted 
nine  SCE  site  visits.  Another  agency  estimated  the  single 
acquisition  cost  of  using  SCE  to  be  as  low  as  $20,000.  The  SCE 
traming  cost  can  be  amortized  over  a  larger  number  of  SCEs 
as  the  individual  team  members  participate  on  other  source 
selections  or  acquisitions.  The  agency  as  a  whole  will  benefit 
over  time  by  reducing  training  costs  as  the  numbers  of  trained 
personnel  increase  and  SCE  use  becomes  more  routine. 

Given  a  $10  million  acquisition,  which  was  introduced  earlier 
as  a  reasonable  threshold  for  seriously  considering  the  use  of 
SCE,  and  a  similar  number  of  offerors  as  shown  in  Figure  3-4, 
the  SCE  cost  is  0.6%  of  the  total  program  cost.  While  using 
SCE  to  help  select  the  most  qualified  offeror  will  not  eliminate 
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cost  or  schedule  problems,  it  will  point  out  the  offeror(s)  with 
the  best  proven  practices  to  mitigate  such  problems,  which 
can  more  than  pay  for  itself  over  the  life  of  the  contract. 


Itemized  Expeneee 

Unit  Costs 
(1  week) 

Total  Costs 
(9  site  visits) 

Travel  Expenses  (5  person  team) 

2,500 

22,500 

Hotel 

1,600 

14,400 

Per  Diem  and  Miscellaneous 

1,500 

13,500 

One-time  Training  Course 

11,000 

Total  SCE  Costs 

$61,400 

Figure  3-4  Example  SCE  Cost  Summary  (Training  Plus  On¬ 
site) 


The  costs  of  an  offeror  supporting  an  SCE,  both  in  the  human 
resource  and  financial  terms,  are  significant — though  not 
always  as  high  as  those  of  the  government.  Considerable 
preparation  time  is  expended  by  a  company — ^as  the  company 
is  typically  trying  to  put  its  best  foot  forward  for  the 
government,  especially  if  the  SCE  is  done  in  conjunction  with 
a  source  selection.  Thus,  all  offerors  will  perform  some 
preparatory  tasks  to  accommodate  an  SCE. 

Figure  3-5  provides  an  estimate  of  those  costs,  using  $200,000 
as  the  cost  per  person-year  or  $3,850  per  person-week.  The 
preparation  time  of  four  person-weeks  accounts  for  one 
person  full-time  for  four  weeks  or  two  individuals  working 
full-time  for  two  weeks  prior  to  the  SCE  site  visit.  Activities 
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include  identifying  projects  for  review,  getting  maturity 
questionnaires  and  project  profiles  completed,  and 
coordinating  the  site  visit  activities. 


hemized  Expense* 

Cost 

Preparation  Time  (4  person-weeks) 

15,400 

Site  Visit  Impact  (1  person-week) 

3,850 

Offeror  POC  and  Debriefing  Time  (3  person-weeks) 

11,550 

Total  Offeror  SCE  Cost 

$30,800 

Figure  3-5  Example  SCE-Imposed  Offeror  Costs 


The  site  visit  costs  are  those  associated  with  conducting 
individual  interviews.  The  final  costs  are  those  produced  by 
the  offeror  point  of  contact  (POC),  who  accompanies  the  SCE 
team,  coordinates  activities  with  the  company,  and  schedules 
the  individuals  for  interviews.  This  POC  also  prepares 
individuals  before  each  interview  and  debriefs  the 
interviewee  after  each  interview.  These  costs  vary 
considerably  from  offeror  to  offeror. 

Costs  can  increase  if  some  contractor  staff  must  travel  to 
another  site  to  accommodate  an  SCE.  Sometimes  the  projects 
selected  for  the  evaluation  are  within  a  product  line  and 
division  that  may  be  at  different  locations.  While  the  SCE 
Method  encourages  project  selection  within  the  same 
geographical  location,  this  cannot  always  be  done  because  of 
the  offeror's  organizational  make  up.  Offeror  project 
personnel  traveling  to  accommodate  an  SCE  will  not  only  be 
spending  travel  funds,  their  SCE-associated  labor  costs  will  be 
greater  as  well.  Under  these  circumstances,  the  SCE  team 
should  work  with  the  offeror's  POC  in  an  effort  to  minimize 
impact  on  those  affected  projects. 

The  offeror's  preparatory  costs  are  significant:  for  a  period  of 
at  least  a  week,  the  offeror's  operations  will  be  disrupted  by 
SCE  site  activities,  company  preparation,  and  debriefing 
activities.  These  estimated  costs  wiU  change  depending  on  the 
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contractor,  and  also  as  contractor  familiarity  with  the  SCE 
process  grows  and  preparation  becomes  more  standardized 
by  the  contractor. 

There  are  a  number  of  ways  an  acquisition  organization  can 
optimize  the  resources  required  for  SCE  implementation.  This 
section  will  explore  some  of  the  alteriuitives  available  to 
minimize  the  costs  of  implementing  SCE  in  an  acquisition. 

•  Assign  full-time  SCE  support  to  acquisitions.  This  option 
offers  the  greatest  savings,  in  both  cost  and  personnel. 
Full-time  support  can  take  on  two  levels  of  involvement. 
Personnel  can 

-  Help  with  the  source  selection  documentation  needed  to 
use  SCE,  identify  team  members,  and  coordinate  their 
training. 

-  Augment  the  SCE  teams  for  specific  acquisitions  by 
participating  in  the  on-site  visits. 

Fully  dedicated  personnel,  who  have  already  gone  up  an  SCE 
learning  curve,  should  be  capable  of  implementing  local  SCE 
policies  and  procedures  quickly  and  effectively,  which  should 
reduce  overall  costs. 

The  use  of  one  to  three  full-time  resources  to  augment  a 
particular  program's  SCE  team  will  insure  organizational 
consistency  in  the  practice  of  the  SCE  Method,  and  assist  the 
training  of  personnel  organizationally  through  a  form  of  on- 
the-job  technology  transition  (with  SCE  and  key  software 
development  practices  being  the  technology).  Utilizing  at 
least  one  full-time  resource  will  act  as  a  significant  acquisition 
"force  multiplier"  when  it  comes  to  implementing  SCE  in  an 
organization. 

The  following  approaches  to  cost  reduction  should  be 
avoided  under  all  circumstances  because  they  would  not 
follow  the  SCE  Method. 

•  Site  visit  time  less  than  three  days. 

•  Teams  of  fewer  than  four  people. 
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•  Team  members  untrained. 

•  Using  Maturity  Questionnaire  responses  alone  without 
performing  site  visits. 

•  Accepting  SPA  results  in  lieu  of  conducting  site  visits. 

These  approaches  undermine  the  consistency,  repeatability, 
and  reliability  of  the  SCE  Method, 
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The  previous  chapter  explored  issues  which  are  relevant  to 
making  the  decision  to  use  SCE  in  an  acquisition.  This  chapter 
will  discuss  the  issues  involved  in  actually  using  SCE  in  the 
source  selection  process.  The  first  section  discusses  the  source 
selection  schedule  and  the  implications  of  using  SCE.  The 
next  section  addresses  the  options  for  factoring  the  SCE 
findings  in  the  source  selection  decision. 

This  section  presents  the  source  selection  process  using  the 
RFP  release  point  as  the  date  from  which  to  build  a  source 
selection  schedule.  The  following  subsections  will  examine 
the  SCE  schedule  within  a  source  selection  before  and  after 
RFP  release.  Each  will  present  a  schedule  of  SCE-related 
activities  showing  a  range  of  time  in  which  the  activities  need 
to  be  completed,  not  the  time  to  complete  the  activity.  These 
schedules  are  approximate  rather  than  absolute,  and  should 
be  tailored  to  the  acquisition's  circumstances.  Each  activity  on 
the  schedules  is  annotated  with  a  comment  describing  the 
activity  and  a  number  which  will  be  referenced  in  the  text  for 
fvuther  discussion  of  each  SCE-related  activity. 

The  schedule  presented  in  Figure  4-1  refers  to  the  major  SCE- 
related  source  selection  activities  that  are  typically 
accomplished  before  the  release  of  the  RFP.  The  first  three 
activities — annotated  with  a  "1,"  "2,"  and  "3" — show  start 
dates  in  the  range  of  seven  or  eight  months  prior  to  the  release 
of  the  RFP.  Depending  on  the  acquisition,  these  dates  could  be 
too  close  or  too  far  from  the  target  RFP  release  date.  Activities 
2  through  5  are  explored  in  more  detail  in  Chapter  5. 

Activity  1—SCE  Implementation  Planning.  This  is  the  activity 
discussed  in  Chapter  3 — deciding  to  use  SCE  in  an 
acquisition.  This  activity  could  continue  until  the  day  the 
proposals  are  received,  depending  upon  the  proposed 
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application  of  SCE.  Part  of  the  activity  may  include  inserting 
wording  about  planned  SCE  usage  into  the  Commerce 
Business  Daily  (CBD)  announcement.  This  activity  starts 
before  the  formation  of  an  SCE  team. 

Activities  2  and  3 — Documentation.  This  activity  involves 
preparing  the  documentation  needed  for  the  Source  Selection 
Plan  (SSP)  and  the  Request  for  Proposal  (RFP).  These 
documents  describe  how  SCE  will  be  used  for  the  acquisition, 
the  former  for  the  SSA,  SSAC  and  SSEB,  the  latter  for  the 
offeror  community. 


-7 


‘5 


Month*  LMding  Up  To  RFP  Roleato 


•2 


-1 


SraTTTiFF 

Released 


I  SCE  ImpWmentation  planning 

I  SSP  preparation 


Released 


Evaluation  standard  preparation 

I  I  I  _ 

SCE  team  training  and  preparation  6 

Offerors  prepare  proposals 


Figure  4-1  Sample  SCE  Schedule  Leading  Up  To  RFP  Re¬ 
lease 

Activity  4 — Pre-Proposal  Conference.  This  is  usually  a  one-day 
event  to  present  the  acquisition  strategy,  contract  type, 
evaluation  criteria,  and  major  program  milestones  to 
prospective  offerors.  It  is  an  opportunity  for  the  offeror 
community  to  hear  first-hand  about  the  pending  program  and 
for  the  government  to  receive  feedback  on  the  program  and 
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their  source  selection  approach.  Typically,  a  portion  of  the 
event  will  be  dedicated  to  briefing  how  SCE  will  be  used,  and 
allowing  offerors  to  seek  further  guidance  or  explanation,  if 
needed. 

Activity  5 — Evaluation  Standard  Preparation.  This  activity  is  one 
of  the  most  important  activities  the  SCE  team  leader  or  other 
individual  responsible  will  be  engaged  in  related  to  SCE.  The 
evaluation  standard  will  dictate  to  the  government  team  how 
the  SCE  site  visit  strengths  and  weaknesses  by  key  process 
area  are  measiired  and  then  translated  into  findings  used  in 
the  source  selection  decision.  Standards  are  not  provided  to 
the  offerors. 

Activity  6 — SCE  Team  Training  and  Preparation.  This  activity 
will  vary  in  amoimt  of  work  according  to  the  experience  of  the 
team  and  the  SCE  infrastructure  in  place  at  the  acquisition 
organization  that  supports  the  team.  It  is  recommended  that 
team  training  take  place  within  one  to  two  months  of  the 
actual  site  visits.  If  all  members  of  the  team  have  been  trained, 
but  have  not  worked  together  on  an  SCE,  a  practice  SCE  is 
recommended.  Ail  team  members  should  have  been  trained 
in  SCE  by  the  SEI,  which  is  the  only  official  source  of  traiiung 
at  this  time. 

Figure  4-2  depicts  a  typical  source  selection  schedule  after 
RFP  release.  As  with  previous  schedules,  this  one  is  given  for 
illustrative  purposes  only. 

Activity  1 — Offeror's  Prepare  Proposals.  Within  the  acquisition 
organization,  while  offerors  are  preparing  proposals,  the 
month  after  the  RFP  has  been  released  is  spent  preparing  for 
SCE  site  visits.  During  this  period,  the  SCE  team  should  come 
together  to  prepare  for  the  site  visits,  including  team-building 
activities.  The  offerors  will  have  received  instructions  in  the 
RFP  on  exactly  how  to  prepare  for  the  possibility  of  SCE  site 
visits.  This  will  have  included  specifics  regarding  project 
selection,  responding  to  the  maturity  questionnaire,  and 
establishing  a  point  of  contact  who  will  help  with  the  logistics 
of  the  site  visit. 
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Activity  2 — Initial  Evaluations.  After  receipt  of  the  proposals, 
the  technical,  cost,  and  past  performance  (Performance  Risk 
Analysis  Group /PRAG)  or  other  evaluation  teams  begin  their 
activities.  The  SCE  team  should  be  part  of  the  technical  team 
and  should  also  evaluate  written  proposals  in  software 
area(s).  A  month  is  shown  as  an  estimate.  This  activity  could 
last  more  or  less  time.  The  receipt  of  proposals  is  typically  the 
initiation  of  the  formal  preparation  for  on-site  visits  to  the 
offerors;  however,  the  visits  themselves  will  not  be  conducted 
until  after  the  competitive  range  determination  is  made.  The 
particular  circumstances  of  the  acquisition(e.g.  number  of 
offerors,  SCE  team  availability,  competitive  range 
determination)  will  dictate  the  exact  activities  that  occur  for 
the  SCE  team  during  this  period  of  time. 

Activity  3 — ^SCE  Site  Visits.  The  SCE  team  will  perform  site 
visits  with  all  the  offerors  remaining  in  the  competitive  range. 
Precisely  when  the  SCEs  are  to  be  conducted  is  largely 
dictated  by  how  SCE  is  being  used  in  contributing  to  the 
source  selection  decision  as  described  in  the  SSP  or  evaluation 
plan.  For  instance,  if  the  SCE  results  will  be  factored  into  an 
item  or  items  of  specific  criteria,  they  should  be  conducted 
after  receipt  of  CR/DR  resporises  but  prior  to  face-to-face 
discussions.  If  SCE  is  to  be  used  to  support  PRAG  (past 
performance)  findings,  then  site  visits  can  be  accomplished 
anytime  after  competitive  range  determination  but  before 
Best  and  Final  Offers  (BAFOs)  are  issued. 

Activity  4 — Discussions  /  Negotiations.  This  activity  addresses 
that  part  of  the  process  where  Clarification  Requests  (CRs), 
Deficiency  Reports  (DRs),  and  Points  For  Negotiation  (PFNs) 
are  communicated  to  the  offerors  within  the  competitive 
range.  These  tools  can  also  be  used  by  the  government  to 
communicate  SCE  findings  to  the  offerors.  The  government 
then  allows  all  remaining  offerors  to  respond  to  each  and  then 
requests  these  offerors  to  submit  a  BAFO.  The  goverrunent 
will  also  begin  developing  "Model"  contracts  for  those 
offerors  stiU  within  the  competitive  range  to  reflect  the  terms 
and  conditions  agreed  upon  by  both  parties  (the  government 
and  that  particular  offeror).  Offerors  are  advised  that  any 
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deviation  from  the  agreed  terms  and  conditions  that  are  not 
traceable  from  their  original  proposal  may  result  in  their 
proposal  being  considered  unacceptable.  This  period,  too,  can 
last  more  or  less  time  than  depicted  in  Figure  4-2. 


1 


Months  after  RFP  Release 
2  3  4 


Proposals 

received 


Offerors  prepare  proposals 

2  Initial  evaluations 


SCE  site 


visit 

Discussions/negotiations 

BAFO  evJuations 

Source  selection  decision  3 


Figure  4-2  Sample  SCE  Schedule  After  RFP  Release 


Activity  5 — BAFO  Evaluations.  Here,  the  government 
considers  the  offerors'  BAFOs.  This  may  entail  significant 
analysis  comparing  the  offeror's  responses  as  to  their  effect 
upon  the  analysis  and  determinations  formulated  up  to  this 
point.  Here  again  the  new  or  revised  information  is  analyzed/ 
evaluated  against  the  approved  evaluation  standards  used  in 
evaluating  the  offerors  iiutial  proposal. 

Activity  6 — Source  Selection  Decision.  Once  all  of  the 
aforementioned  activities  have  been  completed,  an  award 
decision  will  be  made.  The  SSA  will  have  been  convinced  that 
an  equitable,  effective  and  objective  evaluation  of  each 
offeror's  strengths,  weaknesses,  and  improvement  activities 
has  been  made  by  the  SSEB/T.  The  time  from  receipt  of 
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BAFOs  to  contract  award  can  take  some  time  as  there  are  a 
considerable  number  of  legally  imposed  actions  that  must 
take  place  before  announcing  an  award. 

This  section  explores  using  the  findings  from  an  SCE  site  visit 
in  the  final  decision  of  a  source  selection.  Section  M  of  the  RFP 
notifies  offerors  that  the  use  of  SCE  would  be  evaluated  as  a 
specific  criterion.  Included  in  this  section  will  be  an  example 
using  a  color  code  scheme  as  the  rating  tool  in  the  sovirce 
selection  process.  The  discussions  that  follow,  while  using 
data  from  real  acquisitions,  have  been  edited  to  eliminate 
source  selection  sensitivity  or  to  illustrate  a  key  p>oint  about 
SCE  implementation.  A  reference  to  the  Performance  Risk 
Analysis  Group  (PRAG)  is  included  at  the  end  of  this  section. 
SCE  findings  v/ill  be  incorporated  into  the  performance  risk 
assessment  report/ briefing.  There  is  a  significant  difference 
between  specific  criteria  and  performance  risk  assessments. 
The  source  selection-related  regulations,  regardless  of  the 
implementing  agency,  require  tixat  specific  criteria  encompass 
the  characteristics  of  the  program  being  acquired.  All 
acquisition  agencies  require  the  establishment  of  order  of 
precedence  for  the  various  specific  criteria,  so  that  the  offerors 
understand  their  relative  importance  and  can  craft  their 
proposal  accordingly. 

Additionally,  pre-estabUshed  standards  of  evaluation  are 
prepared  for  each  criterion  and  the  offerors'  proposal  is 
measured  against  those  standards  by  the  SSEB.  This 
evaluation  against  the  evaluation  standards  then  forms  the 
basis  of  comparison  of  one  proposal  to  another,  which  is  done 
in  a  source  selection,  typically  by  a  more  senior  body,  such  as 
the  Source  Selection  Advisory  Council. 

Note  that  in  developing  any  evaluation  standards  (Figure  4-4 
and  Figure  4-5),  the  appropriate  procurement  regxilations 
should  be  followed  as  well  as  consulting  and  working  with 
the  source  selection  and  PK  staffs. 

To  get  the  most  emphasis  of  SCE  use  in  source  selection,  SCE 
should  be  used  as  a  specific  criterion  and  may  also  be 
evaluated  by  the  PRAG  for  performance  risk.  Use  of  SCE 
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results  as  specific  criterion  and/or  in  the  PRAG  for 
performance  risk  will  be  decided  by  the  SSA  at  the  same  time 
the  SSP  is  approved,  based  on  source  selection  regulations 
and  program  requirements. 


Each  offeror's  proposal  will  be  evaluated  against  the 
following  areas  listed  in  descending  order  of  importance:  (List 
areas  in  descending  order  of  importance  or  specify  relative 
importance.  (Note:  Areas  should  be  limited  to  two  (including 
cost/price),  when  feasible).  The  technical  area  (or  each  of  the 
areas  [except  cost/ price]  if  more  than  two  areas  used)  will  be 
rated  in  three  ways:  a  color/ adjectival  rating,  a  proposal  risk 
rating  and  a  performance  risk  rating.  The  color/adjectival 
rating  depicts  how  well  the  offeror's  proposal  meets  the 
evaluation  standards  and  solicitation  requirements.  Proposal 
risk  assesses  the  risk  associated  with  the  offeror's  proposed 
approach  as  it  relates  to  accomplishing  the  requirements  of 
this  solicitation.  Performance  risk  assesses  the  probability  of 
the  offeror  successfully  accomplishing  the  proposed  effort 
based  on  the  offeror's  demonstrated  present  and  past 
performance.  The  governir  ^nt  will  conduct  a  performance 
risk  assessment  based  on  the  offeror's  relevant  present  and 
past  perforr  lance.  In  assessing  this  risk,  the  goverrunent  will 
use  performance  data  to  evaluate  the  areas  listed  above. 

Offerors  are  to  note  that  in  conducting  the  performance  risk 
assessment,  the  Goverrunent  will  use  both  data  provided  by 
the  offeror  and  data  obtained  from  other  sources.  Within  each 
area  (other  than  cost/price),  each  of  the  three  ratings — color/ 
adjectival,  proposal  risk  and  performance  risk — will  be 
considered  in  making  an  integrated  source  selection  decision 
as  to  which  proposal  is  most  advantageous  to  the 
government. 

SCE  should  be  used  as  an  item  under  an  area  of  specific 
criterion  such  as  Technical/Nianagement  and/or  in  the 
PRAG  for  performance  risk  assessments.  Ultimately,  how 
SCE  findings  are  translated  into  SCE  results  and  used  in  the 
Source  Selection  (SS)  should  be  determined  by  the  SSA  based 
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cn  source  selection  regulations  and  program  requirements. 
Figure  4-3  provides  an  illustration  from  an  acquisition 
employing  SCE  as  a  technical  item  {software  engineering 
capability)  in  the  technical  area. 


Technical  Area 

Deacription 

Software  Engineering  Capability 
Item 

The  government  will  evaluate  the  offeror's  management  approach  to 
accomplish  contract  goals  and  the  extent  to  which  the  approach 
achieves  the  objectives,  goals,  and  requirements  of  the  solicitation. 
The  government  will  focus  on... 

Technical  Approach  Hem 

The  government  will  evaluate  the  offeror’s  technical  approach  to 
accomplishing  the...  tasks.  The  evaluation  will  assess  the  extent  that 
the  approach  satisfies  the  objective.,  goals,  and  requirements  of  the 
Statement  of  Work. 

Management  Kern 

The  government  will  evaluate  the  offeror's  software  process  by 
reviewing  its  Software  Process  Improvement  Plan  (SPIP)  and  by 
conducting  an  on-site  visit  using  the  Software  Engineering  Institute- 
(SEI)  developed  Software  Capability  Evaluation  (SCE)  Method. 

Figure  4*3  Sample  Set  of  Specific  Criteria  or  Technical 
Items 


What  follows  is  an  example  using  SCE  as  a  specific  criterion 
in  making  the  source  selection  decision.  The  specific  needs  of 
the  acquisition  should  dictate  the  exact  approach  to  be  used. 
In  this  example,  the  items  of  the  Technical  Area  are  listed  in 
descending  order  of  importance.  This  example  is  but  one 
approach  and  method  for  implementing  SCE  findings  in  the 
source  selection  decision. 

This  example  continues  the  discussion  of  SCE  as  a  specific 
criterion  as  depicted  in  Figure  4-3.  The  example  will  illustrate 
the  incorporation  of  the  SCE  findings  into  the  various  source 
selection  evaluation  tools /documents  that  are  used  for  the 
sovirce  selection  as  well  as  the  definitions  established  for  the 
various  color  ratings  and  the  identification  of  risk. 
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Color  Coding  the  Applying  color  codes  begins  when  the  SCE  team  has 

Technical  Items  completed  all  site  visits  and  the  evaluations  of  the  offerors' 

Software  Process  Improvement  Plans  (SPIP).  In  this  example 
the  SPEP  was  requested  to  be  prepared  and  submitted 
separately  at  the  same  time  the  proposal  was  submitted. 

Using  this  approach,  each  technical  item  is  assigned  a  color 
which  corresponds  to  a  rating — from  "exceptional"  to 
"unacceptable."  For  each  item,  an  evaluation  standard  is 
written  to  define  precisely  what  an  offeror  must  do  to  be 
assigned  a  certain  color. 

Figure  4-4  shows  how  colors  were  used,  and  how  ratings  such 
as  "exceptional"  were  defined  [USAF  88]. 


Color 

Rating 

Definition 

Blue  (B) 

Exceptional 

Exceeds  specified  performance  of  capability  in  a 
beneficial  way  to  the  government.  Has  high 
probability  of  satisfying  the  requirement.  Has  no 
significant  weakness. 

Green  (G) 

Acceptable 

Meets  evaluation  standards,  has  good  probability 
of  satisfying  the  requirermnt.  Any  weakness  can  be 
readily  corrected. 

Yellow  (Y) 

Marginal 

Fails  to  meet  evaluation  standards  or  has  tow 
probability  of  meeting  the  requirement;  or  has 
significant  but  correctable  deficiencies. 

Red(R) 

Unacceptable 

Fails  to  meet  a  minimum  requirement.  Deficiency 
requires  a  major  revision  to  correct. 

Figure  4-4  Description  of  Colors 

Along  with  each  color,  the  evaluation  team  assigns  a  risk 
rating  which  reflects  the  risk  associated  with  the  offeror 
performing  on  time,  cost,  and  within  the  specified 
performance  parameters.  Figure  4-5  lists  the  ratings  and  their 
definitions.  This  example  used  the  consistency  between  the 
SCE  findings  and  the  achievability  of  the  offeror's  software 
process  improvement  program  to  denote  the  risk  of  the  item. 
Software  Engineering  Capability. 
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Risk 

Definition 

High  (H) 

Likely  to  cause  significant  serious  disruptior.  of 
schedule.  Increase  in  cost,  or  degradation  of 
performance  even  with  special  contractor  emphasis 
and  close  government  monitoring. 

Moderate  (M) 

Can  potentially  cause  some  disruption  of  schedule, 
increase  in  cost,  or  degradation  of  performance. 
However,  special  contractor  emphasis  and  close 
government  monitoring  will  probably  be  able  to 
overcome  difficulties. 

Low  (L) 

Has  little  potential  to  cause  disruption  of  schedule, 
increase  in  cost,  or  degradation  of  performance. 
Normal  contractor  emphasis  and  government 
monitoring  will  probably  be  able  to  overcome 
difficulties. 

Figure  4-5  Description  of  Risks 

A  complete  set  of  findings  (documented  for  each  KPA)  on  an 
offeror's  strengths  and  weaknesses,  measvired  against  the 
CMM  KPAs,  should  be  used  in  assigning  color  codes  and 
risks.  The  SCE  team  should  provide  the  SSEB  with  these 
findings.  See  Figure  4-6,  Figure  4-7,  and  Figure  4-8  for  an 
example.  (Figure  4-6  is  a  summary  of  the  findings,  while 
Figure  4-7  and  Figure  4-8  show  the  details  of  that  summary.) 


Summary  Reaulta 

Strong 

•  Requirements  Management 

•  Peer  Reviews 

•  Software  Project  Tracking  and  Oversight 

Acceptable 

•  Software  Project  Planning 

•  Software  Configuration  Management 

•  Software  Quality  Assurance 

•  Training  Program 

Weak 

•  Organization  Process  Focus 


Figure  4-6  Summary  Findings  From  a  Recent  SCE 
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The  source  selection  organization  should  at  no  time  ask  for  or 
accept  findings  from  a  Software  Process  Assessment  (SPA). 
As  discussed  previously,  in  Chapter  1,  SPA  findings  are 
determined  for  a  different  purpose  and  are  inappropriate  for 
use  with  SCE  findings  in  a  source  selection  decision. 

The  summary  findings  shown  in  Figiire  4-6  reveal  that  only 
one  key  process  area  was  weak.  The  weaknesses  contributing 
to  that  determination  can  be  found  in  Figure  4-7  and  Figure  4- 
8.  Although  there  were  weaknesses  in  other  key  process  areas, 
only  the  Organization  Process  Focus  weaknesses  were  found 
to  be  significant  enough  for  that  KPA  to  be  included  in  the 
summary  findings  weak  area.  The  details  of  that 
detei^nination  are  made  by  the  SCE  team  in  the  context  of  this 
specific  acquisition.  This  means  that  the  SCE  team  used  their 
individual  professional  judgment  to  determine  the  degree  of 
satisfaction  of  the  goals  of  each  KPA.  The  context  of  these 
determinations  is  critical  to  the  findings.  For  example,  it  is 
possible  that  a  software  configuration  maruigement  system 
covild  exist  in  an  organization  that  by  most  experienced 
persormel's  viewpoint  would  be  considered  excellent. 
However,  the  SCE  team  may  have  found  that  one  project  does 
not  vise  it,  another  project  uses  it  very  effectively,  and  a  third 
or  fourth  project  may  use  it  in  differing  levels  of  applicatioru 
’fhis  is  an  example  where  the  SCE  team  would  be  faced  with 
determining,  from  the  organizational  standpoint,  whether  a 
finding  for  the  Software  Configuration  Management  KPA  is 
acceptable,  weak  or  strong.  On  one  hand  it  was  determined 
that  the  configuration  management  system  in  place  is 
excellent  (a  strength),  on  the  other  hand  the  evidence  suggests 
spotty  implementation  and  or  application  (acceptable  or 
weak?).  Does  this  mean  the  finding  is  reported  as  a  strength, 
acceptable  or  as  a  weakness?  This  type  of  dilemma  is  typical 
of  those  faced  by  the  SCE  team  for  which  the  various 
background  experience  in  the  different  disciplines  comes  into 
play  in  providing  consensus  from  a  professional  judgment 
standpoint  on  specific  findings  for  each  KPA  investigated. 
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Requirements  Management 
Strengths 

•  Effective  review/statusing  mechanism  in  place 

•  Very  strong,  clear  lines  of  authority 

•  Software  engineering  process  represented 
throughout  system  engineering  process  (and 
vice  versa) 

•  Action  items  tracked  to  closure  by  management 

•  Sure  technical  presence  at  managerial  level 

Weaknesses 


Improvement  Activities 
none  noted 


Peer  Reviews 

Strengths 

•  Multiple,  rigorous  requirements,  design,  and 
code  inspections  conducted 

•  Training  required  to  participate  on  peer  reviews 

•  Experienced,  senior  people  lead  reviews 

•  Currently  tracing  defects  and  beginning  to 
analyze  results 

Weaknesses 

•  Lack  of  organizational  consistency  in  the 
reviews  of  each  phase 


improvement  Activities 
none  noted 


Software  Project  Tracking  and  Oversight 

Strengths 

•  Provides  wide  coverage  of  software  process  at 
a  detailed  level 

•  Extensive  use  of  programmers'  notebooks  to 
guide  staff  through  various  phases  of  the  pro¬ 
cess 

•  Errphasis  on  populating  useful  software  devel¬ 
opment  folders 

Weaknesses 

•  Lack  of  organizational  consistency 

•  Inadequate  resources  to  timely  train 

improvement  Activities 

none  noted 


Software  Quaiity  Assurance 
Strengths 

•  Experienced  personnel 

•  Very  good  relationship  with  development  per¬ 
sonnel 

•  Independent  reporting  chain 

Weaknesses 

•  Lack  of  sampling  mechanism 

•  Lack  of  independent  audit  coverage  and  depth 

•  Resources  lacking  on  some  projects 

improvement  Activities 

•  Establishing  an  independent  reporting  chain 

•  Interviewing  for  SQA  personnel 


Figure  4*7  Detailed  Findings 
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Figure  4-8  Detailed  Findings  (continued) 

Another  aspect  of  using  SCE  is  illustrated  by  the  use  of  Points 
for  Negotiation  (PFNs)  to  communicate  software  process 
capability  weaknesses  identified  by  SCE  to  the  offerors  within 
the  competitive  range.  A  Clarification  Request  (CR)  should  be 
used  to  communicate  a  weakness  irutially.  A  PEN  can  be  used 
to  identify  those  points  the  government  wishes  to  discuss 
further.  A  PFN  or  CR  wiU  never  be  used  to  identify  a 
deficiency.  The  SSEB  then  considered  their  responses  with  the 
original  SCE  findings  before  making  a  final  determination 
against  the  evaluation  standard.  This  approach  allows  the 
offerors  the  opportunity  to  point  out  any  oversights  on  the 
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part  of  the  SCE  team.  The  SCE  team  could  prepare  a  PEN  (or 
CR  if  appropriate)  to  let  offerors  know  what  weaknesses  were 
found.  Figure  4-9  is  an  example  of  a  PFN.  This  example  details 
the  specific  weaknesses  found  by  the  SCE  team  that  made  the 
KPA  organization  process  focus  weak. 


Source  Selection  Information  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


Government  Reference: 
IFPP  Paragraph  3.3.4 


Offeror  Reference; 


Point  for  Negotiation; 


POINT  FOR  NEGOTIATION 


Offeror; 

XYZ  Corporation 


Register  Number; 
PFN-XYZ-S-001 


The  key  process  area  (KPA)  found  by  the  Software  Capability  Evaluation 
(SCE)  to  be  weak  is  Organization  Process  Focus.  The  detailed  findings  leading 
to  this  conclusion  are  as  follows: 

•  Lack  of  buy  in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


SCE  team  Chief: 


SSEB/T  Chairperson: 


Source  Selection  Information  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


Figure  4-9  Findings  Incorporated  Into  a  Point  For  Negotia¬ 
tion  (PFN) 


The  findings  that  go  into  a  PFN  may  vary.  One  acquisition 
organization's  approach  was  to  provide  only  those 
weaknesses  in  the  PFN  that  caused  an  entire  key  process  area 
to  be  evaluated  as  weak  (as  in  Figure  4-8).  Those  are 
significant  weaknesses  which,  depending  upon  the  affected 
key  process  area,  may  influence  the  evaluation  standard  one 
way  or  another.  Alternatively,  the  entire  findings  set  may  be 
communicated  in  the  PFN.  In  deciding  what  to  include  in  the 
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PFN/CR,  the  SCE  team  leader  should  work  very  closely  with 
the  PCO,  SSEB  chairperson,  and  the  acquisition 
organization's  legal  advisor. 

A  PFN/CR  is  a  way  to  communicate  an  SCE  weakness(es)  to 
an  offeror  and  allow  the  offeror  to  respond  with  one  of  the 
following; 

•  Evidence  showing  the  goverrunent's  SCE  team  made  an 
oversight. 

•  A  response  accepting  the  findings. 

•  A  response  accepting  the  findings  and  identifying  im¬ 
provement  activities  to  remedy  the  weaknesses. 

•  A  combination  of  the  above  previous  responses. 

A  cover  letter  sent  with  the  PFNs/CRs  will  explain  how  the 
offeror  may  respond.  It  is  recommended  that  the  letter 
include  a  page  limitation  for  the  offeror's  response  so  that  the 
SSEB  is  provided  with  only  relevant  evidence. 

When  the  responses  to  the  PFNs/CRs  have  been  received 
from  the  offerors  (typically  five  to  seven  days  are  allowed  for 
responses)  the  SCE  team  leader  should  analyze  them  to  see  if 
material  changes  in  the  findings  are  required  that  would 
necessitate  recalling  the  SCE  team.  The  or\ly  time  the  SCE 
team  would  reverse  a  decision  on  a  finding,  is  if  the  offeror 
shows  proof  that  the  team  overlooked  something. 

The  SCE  team  performs  an  analysis  and  makes  any  final 
adjustments  to  the  findings.  These  findings  will  be  factored 
into  the  technical  area/item  evaluation  results  for  each 
offeror.  The  manner  in  which  SCE  findings /results  are 
factored  into  the  source  selection  results  depends  upon  how 
SCE  was  structured  into  the  sovirce  selection  (e.g.  items, 
factors  etc.)  Your  PCO  and  procurement  regulations  will 
guide  you  through  this  step.  Figure  4-10  and  Figure  4-11 
provide  an  example  item  summary  for  the  set  of  findings 
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shown  in  Figvire  4-6,  Figxire  4-7,  and  Figure  4-8.  The  example 
assumes  that  no  changes  to  the  findings  were  made  during 
the  PFN/CR  process. 

The  item  summary  contains  the  color  rating  and  associated 
risk  for  the  respective  offer,  some  background  on  the  projects 
the  SCE  evaluated,  the  summary  and  detailed  findings  made 
by  the  SCE  team  while  on  site,  and  a  statement  justifying  the 
assigned  risk.  In  order  to  determine  the  color  rating,  the  SCE 
team  applied  the  findings  to  the  evaluation  standard. 
Similarly,  for  this  example,  the  risk  was  assigned  based  upon 
consistency  between  the  offeror's  communicated  capability 
found  in  the  SPIP  and  the  actual  SCE  findings.  In  this 
example,  the  offeror  was  rated  blue  with  a  low  risk.  The  item 
summary  then  points  out  the  various  strengths  and 
weaknesses  in  their  appropriate  location  and  justifies  the  risk 
rating. 

At  this  level  of  evaluation,  within  the  SSEB,  the  offerors  are 
only  compared  to  a  pre-established  standard.  No  offerors  are 
compared  to  one  another. 
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Source  Selection  Information  (See  FAR  3.104}  —  For  Official  Use  Only  (When  Riled  in) 

Item  Summary 

Area:  Technical  Item:  T3/Software  Offeror:  XY2  Corp  Color  Rating:  Blue 

Engineering  Capability 

Description  of  Proposal 

The  offeror  proposed  a  software  PIP  which... 

The  software  Process  Improvement  Plan  was  found  to  be  consistent  with  the  SCE  findings.  The  offeror’s 
program  of  software  process  improvement  is  genuine,  with  considerable  emphasis  on  organizational 
standardization  and  removal  of  defects  through  rigorous  reviews.  The  projects  examined  as  part  of  the 
Software  Capability  Evaluation  (SCE)  are  as  follows: 

.  ABCD  •  HAVE  GOLD  PLATE  •  COBRA  LIBRARY  •  CCXYZ 


Strengths 

Requirements  Management 

•  Defined  review/status  mechanism  in  place 

•  Very  clear,  strong  lines  of  authority 

•  Software  engineering  represented  throughout  system  engineering  process  (and  vice  versa) 

•  Action  items  tracked  to  closure  by  management 

•  Sure  technical  presence  at  management  level 

Software  Project  Tracking  and  Oversight 

•  Provides  wide  coverage  of  software  process  at  a  detailed  level 

•  Extensive  use  of  programmers  notebooks  to  guide  staff  through  phases  of  the  process 
■  Firm  emphasis  on  populating  useful  software  development  folders 

Peer  Reviews 

•  Multiple,  rigorous  requirements,  design,  and  code  inspections  conducted 

•  Training  required  to  participate  on  peer  reviews 

•  Experienced,  senior  people  lead  reviews 

•  Currently  tracking  defects  and  beginning  to  analyze  results 

Item  Chief  Signature:  Area  Chief  Signature: 

Figure  4-10  Findings  Incorporated  in  Item  Summary 
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Acceptable  Points 
504 

•  Experienced  personnel  and  independent  reporting  chain 
Software  Project  Planning 

•  Procedure  for  sizing  and  costing  software  exist  project  to  project 

•  Extensive  collection  of  management  metrics  and  tracking  of  progress 

SCM 

•  Effective  change  control  process  and  traceability  between  development  projects 
Training  Program 

•  Solid  emphasis  from  management  and  extensive  in-house  software  courses 

•  SEPG 

•  An  organizational  function  exists  with  full-time  resources  in  place 

Weaknesses 

The  Key  Process  Area  found  by  the  Software  Capability  Evaluation  to  be  weak  was; 
Organization  Process  Focus 

•  Lack  of  buy-in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


Overall  Risk  Assessment  and  Evaluation  Summary 

Low  Risk;  The  consistency  between  their  SCE  findings  and  software  process  improvement  plan  shows 
they  understand  their  current  maturity  level  and  where  they  are  going  as  an  organization.  They  are  very 
strong  technically  (very  close  to  being  strong  in  all  the  key  process  areas)  and  are  committed  to 
developing  quality  software  using  a  continually  improving  development  process.  This  contractor’s 
commitment  to  process  improvement  was  further  evidenced  by  the  process  rigor  in  place  on  one  of  their 
commercial  programs  where  no  development  standards  were  required.  Their  process  was  still  the  same 
and  management  exercised  the  same  controls. 


Figure  4-1 1  Findings  Incorporated  in  Item  Summary 
(continued) 

Item  Summaries  are  reviewed  by  the  SSEB/T  chairperson  and 
then  an  area  summary  is  prepared  which  normally  "rolls  up" 
all  (or  most)  of  the  strengths  and  weaknesses  from  the 
individual  item  summaries  and  then  identifies  an  overall  risk 
for  that  area.  This  information  is  reviewed  by  the  PCO,  legal 
representatives,  and  the  SSAC.  The  legal  and  rco  review  will 
examine  everything  to  insure  that  the  evaluation  standards 
have  been  consistently  applied  and  that  the  item  and  area 
summaries  contain  consistent  types  and  levels  of  information. 
The  SSEB/T  will  present  this  information  to  the  SSAC.  The 
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SSAC  members  will  analyze  the  SSEB/T's  evaluation  results 
and  start  the  process  of  comparing  each  offerors  strengths, 
weaknesses  and  risk — an  activity  the  SSEB  is  not  allowed  to 
do. 

In  parallel,  the  SSEB  will  make  a  formal  presentation  to  the 
SSAC  outlining  the  color  codes,  strengths,  weaknesses  and 
risks  for  each  offeror  for  each  item  and  area  resulting  from 
their  evaluations.  During  this  presentation,  the  SCE  team 
leader,  as  a  member  of  the  SSEB,  should  be  prepared  to 
elaborate  on  any  of  the  findings  from  any  of  the  offerors.  For 
example,  the  SCE  team  leader  should  be  prepared  to  explain 
not  only  why  an  offeror  was  weak  in  software  configuration 
management,  but  also  why  the  SCE  team  found  their  change 
control  process  lacking.  The  SSAC  will  want  to  ensure  that  the 
SSEB  can  substantiate  their  findings  with  documented 
evidence. 

At  this  point  in  the  source  selection  process,  the  SSAC,  after 
completing  their  comparative  analysis  of  all  final  proposals' 
strengths,  weaknesses  and  risks,  may  elect  to  assign  a 
different  color  code  separate  from  the  SSEB  or  it  may  ask  the 
SSEB  to  reconsider  its  color  codes  in  light  of  i^iformation 
discussed  in  the  SSAC  briefing.  These  actions  are  normally 
done  on  an  "exception"  basis  and  a  'e  not  common  since  the 
SSAC  would  have  reviewed  this  material  at  the  time  of 
competitive  range  and  before  BAFOs  were  issued,  therefore, 
any  "disconnects"  should  be  resolved  before  BAFOs  are 
received.  Unless  an  offeror  completely  changes  its  proposal 
approach,  there  should  be  no  "surprises"  in  the  BAFOs.  The 
SSAC  will  ensure  that  the  evaluation  for  each  criterion  has 
been  consistently  and  fairly  applied  to  all  offerors. 

Figure  4-12  shows  one  way  the  findings  from  a  series  of  SCEs 
has  been  presented  formally  to  an  SSAC.  Each  offeror's 
technical  rating,  strengths  and  weaknesses,  risk,  and  a 
summary  explaining  the  basis  for  the  risk  are  identified  and 
placed  next  to  the  other  offerors  so  that  the  SSAC  may 
compare  and  discuss  them  during  a  presentation.  This 
normally  represents  the  lowest  level  of  detail  presented  to  the 
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SSAC  during  the  formal  presentation.  It  is  during  this 
presentation  that  a  SCE  team  leader  may  have  to  articulate 
why  certain  key  process  areas  were  a  strength  or  weakness  for 
a  particular  offeror.  The  expertise  of  the  SCE  team  leader  is 
needed  to  communicate  why  a  KPA  was  strong  or  weak  and 
its  significance  within  the  software  process. 


Item:  T-3  Software  Engineering  Capability 

Offeror  A 

Offeror  B 

Offeror  C 

Blue 

Yellow 

Yellow 

Strength 

•  Requirements  Manage¬ 
ment 

•  Peer  Reviews 

•  Software  Project  Tracing 
and  oversight 

'  None 

•  Organization  Process 

Focus 

Weakneee 

•  Software  Engineering 
Process  Group 

•  Organization  Process 

Focus 

•  Software  Quality 

Assurance 

•  Training  Program 

•  Peer  Reviews 

•  Software  Project  Tracking 
and  Oversight 

•  Training  Program 

Riek 

. .  1 

Offeror  is  very  strong 
technically  and  is  committed 
to  developing  quality 
software  using  a 
continuously  improving 
development  process 

Because  of  the  large 
disparity  between  our 
findings  and  their  submitted 
SPIP,  it  is  highly  questionable 
whether  the  software  process 
improvement  is  being 
implemented 

Offer  has  a  realistic  SPIP 
indicating  they  are  at  the 
initial  maturity  level  with  their 
best  practices  being  applied 
to  all  new  prog-^ams 

L 

H 

M 

Figure  4*1 2  Findings  Output  From  the  Evaluation  Standard 


The  SCE  written  report  must  also  back  up  and  provide 
substantiation  or  articvilatc  reasons  for  the  ratings'  assigned 
since  the  briefing  is  reduced  to  "bullets"  only  and  should  be 
derived  from  the  detailed  written  findings. 

Figure  4-12  illustrates  how  risk  was  assigned  to  the  software 
engineering  capability  techiucal  score  (color  rating)  in  a  recent 
source  selection.  Note  that  Offerors  B  and  C  have  yellow  as 
their  technical  score,  but  Offeror  B  has  a  high  risk  and  O ''■‘ror 
C  has  a  moderate  risk;  yet  Offeror  C  has  three  weak  Key 
Process  Areas  and  Offeror  B  has  only  two.  How  did  this 
occur? 
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Risk  in  this  acquisition  was  assigned  based  upon  the 
consistency  of  the  organization's  process  improvement 
program  and  the  SCE  findings,  because  it  was  stated  clearly 
in  the  RFP  for  this  acquisition  that  an  organization  could  be  at 
the  Initial  maturity  level  and  still  be  awarded  the  contract.  It 
was  also  stated  in  the  Instructions  for  Proposal  Preparation 
(IFPP)  that  risk  would  be  used  as  a  measure  of  an 
organization's  process  improvement  realism.  If  an 
organization  had  a  realistic  program  of  software  process 
improvement,  then  they  were  considered  low  risk,  regardless 
of  their  current  maturity  level  rating.  If  an  offeror  claimed  to 
be  at  the  Defined  or  Managed  maturity  level  in  its  SPIP,  but 
the  SCE  findings  showed  the  offeror  to  be  at  the  Initial  or 
Repeatable  maturity  level,  then  the  SSEB  would  assign  either 
a  high  or  moderate  risk.  This  assignment  depended  upon  the 
magnitude  of  the  disparity  between  the  SPIP  and  the  actual 
SCE  findings. 

Offeror  B  had  identified  themselves  as  being  at  the  Defined 
maturity  level  and  did  not  have  an  improvement  plan  that 
would  substantiate  their  progress  through  the  lower  maturity 
level.  The  SSEB/SCE  team  determined  Offeror  B  to  be  closer 
to  the  Initial  maturity  level.  In  short.  Offeror  B  was  unaware 
of  its  actual  lower  maturity  level  and  was  consequently 
assigned  a  high  risk  with  only  two  weak  key  process  areas, 
while  Offeror  C  received  a  moderate  risk  with  three.  Offeror 
C,  on  the  other  hand,  had  a  realistic  SPIP  indicating  they  were 
at  the  Initial  maturity  level  with  their  best  practices  being 
applied  to  all  new  programs.  The  SCE  findings  confirmed  this 
and  resulted  in  assigning  a  lower  risk  to  this  offeror. 
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Area;  Technical 

Offeror 

A 

Offeror 

B 

Offeror 

C 

Software  Engineering 

Color 

Blue 

Yellow 

Yellow 

capMiiity 

Risk 

L 

H 

M 

Technical  App. 

Color 

Green 

Ye'.low 

Blue 

Risk 

L 

L 

L 

Management 

Color 

Green 

Green 

Blue 

Risk 

L 

L 

L 

SUMMARY  RESULTS 

Color 

Green 

Yellow 

Green 

Riak 

L 

H 

M 

Figure  4*13  Technical  Area  Summary 

The  last  step  of  the  process  is  the  integration  of  the  SCE 
technical  rating  and  risk  factor  with  those  of  the  other 
techiucal  items  to  produce  a  technical  area  summary,  as 
shown  in  Figure  4-13.  At  this  point,  the  SSAC  will  integrate 
the  color  codes  and  risk  factors  into  area  summaries  based 
upon  their  own  analysis  and  presents  them  to  the  SS A.  The 
SSAC  then  conducts  a  comparative  analysis  of  all  offerors' 
strengths,  weaknesses  and  risks  as  presented  by  the  SSEB/T 
on  these  item  and  area  summaries  and  presents  it  to  the  SSA. 
Note:  SSAC  does  not  make  written  recommendations  to  the 
SSA.  Note  that  in  this  example  the  items  in  the  Technical  Area, 
Management,  Technical  Approach  and  Software  Engineering 
Capability  are  listed  in  descending  order  of  importance.  This 
illustration  of  risk  identification  and  assessment  is  not  the  sole 
method  for  approaching  the  risk  problem.  Acquisitions 
should  tailor  the  risk  assignment  to  the  specific  needs  of  the 
acquisition. 

Performance 

Risk  Analysis  Offerors  past  and  present  performance  is  evaluated  by  the 

Group  (PRAG)  PRAG.  Their  results  will  be  presented  to  the  SSA  in  the  form 

of  performance  risk. 
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Performance  Risk  Assessment  Definitions: 

High 

Significant  doubt  exists,  based  on  offeror's  performance 
records,  that  he  can  perform  the  proposed  effort. 


Moderate 

Some  doubt  exists,  based  on  the  offeror's  performance 
records,  that  he  can  perform  the  proposed  effort. 


Low 

Little  doubt  exists,  based  on  the  offeror's  performance 
records,  that  he  can  perform  the  proposed  effort. 

N/A 

No  performance  record  identifiable. 
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Chapter  5  Incorporating  SCE  into  the 

Relevant  Acquisition 
Documentation 

This  chapter  presents  the  major  documents  related  to  the 
source  selection  process  affected  by  the  incorporation  of  SCE. 
The  documents  examined  in  this  chapter  are:  Commerce 
Business  Daily  (CBD)  announcement.  Source  Selection  Plan 
(SSP),  Pre-proposal  Conference  presentation.  Request  For 
Proposal  (RFP),  and  the  Evaluation  Standards.  A  discussion 
accompanies  each  section  describing  why  a  particular 
approach  was  taken.  Each  section  contains  at  least  one 
example  that  can  be  tailored  to  the  unique  needs  of  the 
acquisition. 

Figure  5-1  presents  a  slightly  modified  SCE-related  portion  of 
an  actual  CBD  announcement.  This  announcement  informed 
the  potential  offerors  that 

•  SCE  would  be  performed  to  identify  the  offeror' s  software 
process  capability. 

•  An  offeror's  software  process  capability  would  be  an  inte¬ 
gral  part  of  the  source  selection  decision. 

•  The  government  was  linking  quality,  cost,  and  schedule 
performance  directly  with  software  process  capability. 

•  The  offeror  should  have  a  software  process  improvement 
plan  (SPIP)  in  place  designed  to  achieve  higher  maturity 
levels  of  the  CMM. 

The  message  from  the  government  is  that  offerors  should  be 
implementing  process  improvement  programs  to  achieve 
higher  levels  of  process  capability  and  should  have  a  program 
in  place  to  be  a  "viable"  competitor.  The  RFP  that  would 


Making  the 
Commerce 
Business  Daily 
Announcement 
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follow  a  CBD  announcement  such  as  that  shown  in  Figure  5- 
1  could  reinforce  this  concept  by  requiring  the  submission  of 
a  Software  Process  Improvement  Plan  (SPIP)  as  an  appendix 
to  the  offerors'  proposals. 


The  statement  in  the  CBD,  "Contractors'  software  process 
capability  will  be  verified  by  analyzing  Key  Process  Areas 
(KPAs)  through  the  Defined  maturity  level,  and  a 
demorwtrable  software  process  improvement  program 
leading  to  levels  of  process  capability  higher  than  the  current 
capability"  makes  it  clear  that  the  Defined  maturity  level  is 
not  a  contract  requirement.  Rather,  it  is  a  standard  by  which 
the  evaluation  will  be  conducted,  and  the  source  selection  will 
consider.  It  essentially  defines  the  target  process  capability, 
which  is  the  capability  the  acquirer  is  seeking  for  this 
particular  acquisition  program. 


As  part  of  <AcquisKion  Agency's>  overall  effort  to  improve  software  quality,  cost,  and  schedule 
performance,  the  software  process  capability  of  the  responding  offerers  will  be  a  consideration  in  the 
source  selection  decision.  <Acqui8itionAgency>  will  use  the  Software  Capability  Evaluation  (SCE) 
method  developed  by  the  Software  Engineering  Institute  (SEI)  to  identify  the  current  software 
process  capability  of  responding  offerors  (see  Capability  Maturity  Modal  for  Software  (CMU/SEI-9f  - 
TR-24,  Number  ADA  24603,  August  1991),  Offerors  who  are  determined  to  be  in  the  competitive 
range  after  initial  government  evaluation  of  proposals  is  completed  will  have  their  software  process 
capability  verified  by  an  SCE  team  (via  a  site  visit)  analyzing  key  process  areas  (KPAs)  through  the 
Defined  maturity  level,  and  a  demonstrable  software  process  improvement  program  leading  to  levels 
of  process  capability  higher  than  the  current  capability.  A  conference  will  be  held  at  <time>  on  <date> 
at  <where>  to  answer  any  questions. 


Figure  5-1  Sample  CBD  Announcement 


Why  place  SCE  wording  into  the  CBD  announcement? 
Because  SCE  is  a  relatively  new  technique,  and  not  all  offerors 
will  be  familiar  with  it  or  the  government  use  of  SCE.  Also,  to 
fvirther  define  requirements  so  industry  has  a  better 
understanding  of  the  requirements,  so  companies  who  don't 
meet  the  requirements  won't  waste  bid  and  proposal  money 
on  proposal  preparation.  The  need  to  place  SCE  wording  into 
the  CBD  announcement  may  diminish  in  the  future  after  SCE 
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Placing  SCE  In 
the  Source 
Selection  Plan 


Source  Screening 


use  and  process  improvement  activities  become  routine 
business  practices  throughout  the  acquisition  and  software 
development  communities. 

The  Source  Selection  Plan  (SSP)  outlines  how  the  source 
selection  will  be  conducted  and  subsequently  how  the  award 
decision,  will  be  made.  This  document  is  not  seen  fay  the 
offerors.  SCE  has  a  relatively  small  impact  on  the  production 
of  the  SSP.  Software  Capability  Evaluation  is  discussed  in 
several  places  in  an  SSP.  The  following  subsections  address 
several  of  these. 

In  this  case,  the  government  would  publish  a  sources-sought 
synopsis  in  the  Commerce  Business  Daily  (CBD)  requesting 
that  interested  offerors  provide  to  the  government  their 
qualifications  in  any  one  of  a  number  of  technical  areas 
important  to  the  acquisition.  The  purpose  of  this  activity  is  to 
produce  a  list  of  technically  qualified  offerors.  Maturity  level 
should  never  be  used  as  a  screening  criterion.  However,  the 
presence  of  an  on-going  software  process  improvement 
program  (SPIP)  could  be  used  as  a  screening  criterion. 

Figure  5-2  provides  sample  wording  to  place  SCE  in  the 
Source  Screening  section  of  the  SSP.  The  hypothetical  source 
selection  is  using  Ada  Software,  Radar  Signal  Processing,  and 
Software  Engineering  Capability  as  screening  criteria.  The 
last  statement  in  this  example  communicates  the 
organization's  desire  to  keep  Software  Process  Assessment 
(SPA)  results  out  of  the  source  section  process.  The  SCE  team 
should  not  ask  to  see  SPA  results  when  on  site  (See  Chapter  1 
for  detail  on  the  differences  in  the  two  methods).  Many 
organizations'  process  improvement  programs  can  be 
undermined  by  offerors  trying  to  demonstrate  to  the 
government  a  process  capability  they  carmot  support  on  a 
new  program. 
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3.0  Source  Screening 


Screening  Criteria.  Initial  screening  of  potential  offerors  was  conducted  by  the  publication  of  a 
sources-sought  synopsis  in  the  Commerce  Business  Daily  on  <date>.  The  synopsis  requested 
that  interested  offerors  provide  their  qualifications  in  the  following  areas: 

a.  Software  Engineering  Capability.  Knowledge  of  software  process  improvement  with  a 
verifiable  program  for  process  improvement  using  the  Software  Capability  Evaluation  (SCE) 
Method  developed  by  the  Software  Engineering  Institute  (SEI)  to  identify  the  current  software 
process  capability  of  responding  offerors  (Capability  Maturity  Model  (CMM)  (CMU/SEI-91-TR- 
24,  #  ADA  24603,  Aug  9t))  The  Offeror’s  who  are  determined  to  be  in  the  competitive  range 
after  initial  government  evaluation  of  proposals  is  completed  will  have  their  sofh«vare  process 
capability  verified  by  an  SCE  team  (via  on-site  visit)  analyzing  key  process  areas  (KPAs) 
through  the  Defined  maturity  level,  and  a  demonstratable  software  process  improvement 
program  leading  to  levels  of  process  capability  higher  than  the  current  capability.  Do  not 
provide  Software  Process  Assessment  (SPA)  results. 


SCE  as  a  Specific 
Criterion 


Figure  5-2  SCE  as  a  Screening  Criterion  in  the  SSP 

The  following  example  shows  how  to  use  SCE  as  a  specific  criterion. 
Keep  in  mind  that  this  is  only  an  example.  Each  application  of  SCE 
should  be  tailored  to  accommodate  the  unique  demands  of  the 
acquisition. 

Figure  5-3  provides  a  detailed  description  of  how  a  source  selection 
could  use  SCE  in  the  source  selection  evaluation  process. 
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6.4  Evaluation  Critaria 

6.4.1  Assessment  Criteria 

a.  Soundness  of  approach 

b.  Compliance  with  requirements 

6.4.2  Specific  Criteria 

a.  Technical  Area  -  This  area  will  evaluate  three  rtems  (SEC,  TECH  approach  and 
Management)  represented  here  in  descending  order  of  importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute-  (SEf)  developed  Software  Capability  Evaluation  (SCE).  The 
government  will  determine  the  software  process  capability  by  investigating  the  Key  Process 
Area  (KPAs)  defined  in  the  SEl  Technical  Report,  Capability  Maturity  Model  for  Software 
(CMU/SEI-91-TR-24,  Number  ADA  24603,  August  1991).  The  report  contains  a  description 
of  the  Capability  Maturity  Model  (CMM)  and  the  SEI-defined  maturity  levels.  The  government 
will  perform  an  SCE  of  each  offeror  remaining  in  the  competitive  range  by  reviewing  current 
projects  at  the  site  proposing  on  this  contract  and  comparing  methods/processes  used  on 
those  project  in  the  written  proposal/SPIP. 

The  evaluation  will  result  in  an  organizational  composite,  substantiated  through  individual 
inten/iews  and  reviews  of  documentation,  of  the  offeror's  software  process  activities  on  the 
government-selected  projects.  A  risk  assessment  to  compare  proposed  practices  to  current, 
validated  practices  may  be  performed.  The  evaluation  team  will  determine  findings  of  the 
offeror’s  strengths,  weaknesses,  and  inprovemeht  activities  in  all  KPAs  through  the  Defined 
maturity  level.  Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will  be 
provided  in  writing  during  subsequent  discussions.  The  offeror  will  be  allowed  to  respond  to 
the  findings  with  a  limited  number  of  pages  of  text.  The  on-site  evaluators  may  be  separate 
and  distinct  from  the  proposal  evaluation  team  and  may  include  a  government  contracting 
representative.  All  evaluators  have  been  trained  in  the  SCE  Method. 


Figure  5>3  SCE  as  a  Specific  Criterion  For  The  SSP 

In  Figure  5-3,  the  use  of  SCE  as  a  specific  criterion  falls  under 
one  of  three  items  under  the  technical  area  {SEC,  TECH 
Approach  and  Management)  in  this  case,  it  is  the  most 
important  item.  The  SCE  findings,  in  this  case,  will  form  the 
basis  of  an  item  color  code  and  risk  assessment  and  will  be 
compared  to  an  evaluation  standard  based  on  the  Defined 
maturity  level.  The  SCE  evaluation  is  not  pass/fail — that  is, 
being  less  than  Defined  maturity  level  will  not  exclude  an 
offeror  from  the  competitive  range.  In  this  example,  all 
offerors  will  experience  an  SCE  site  visit  and  be  given  the 
opportunity  to  respond  to  their  evaluated  software  process 
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weaknesses  through  PFNs,  CRs,  Deficiency  Reports  (DRs). 
Complete  SCE  findings  (strengths,  etc.)  should  be  provided  to 
each  offeror  during  the  post  award  debriefings  to 
unsuccessful  offerors,  and  the  post  award  conference  for  the 
winning  contractor(s). 

What  is  presented  in  Figure  5-3  tracks  closely  with  what  is 
presented  in  a  later  section  of  this  chapter,  "Placing  SCE  in  the 
Request  for  Proposal." 

SCE  as  part  of  the  Figure  5-4  shows  an  example  for  using  SCE  findings  with  a 

PRAG.  Note  that  in  this  example  the  SCE  findings  are 
incorporated  with  other  factors  into  the  PRAG  analysis. 


6-3  Present  and  past  performance  (Performance  riak) 

(1 )  Present  and  past  performance  is  a  consideration  in  all  ESC  source  selections. 

Performance  risk  is  a  structured  treatment  of  present  and  past  performance  and  will  be 
determined  for  each  area.  It  is  a  confidence  measure  that  assesses  the  offeror’s  present 
and  past  work  record  in  order  to  determine  the  offeror’s  ability  to  perform  the  proposed 
effort.  Performance  risk  is  assessed  by  the  Performance  Risk  Analysis  Group  (PRAG), 
which  should  be  chaired  by  a  program  manager-level  (senior)  individual  and  consist  of 
government  personnel.  Performance  evaluation  and  risk  assessment  focus  on  relevant 
past  performance  as  it  relates  to  the  specific  criteria.  It  is  the  PRAG’s  responsibility  to 
analyze  the  data  collected;  determine  its  relevance;  and  to  perform  an  “independent"  risk 
assessment.  The  results  of  *he  SCE  will  also  be  factored  into  the  overall  performance  risk 
rating  assigned  by  the  PRAG.  For  information  on  how  to  establish  a  PRAG,  how  it 
operates,  the  forms  which  are  used,  and  how  the  evaluation  is  made  and  reported,  refer  to 
the  ESC  Source  Selection  Handbook. 


Figure  5-4  SCE  as  part  of  the  PRAG  for  SSP 
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Presenting  SCE 
at  the  Pre- 
Proposal 
Conference 


The  pre-proposal  conference  is  an  important  opportunity  for 
the  offerors  to  learn  the  specific,  detailed  requirements  of  the 
contract  and  for  the  government  to  communicate  its 
intentions  and  receive  feedback  from  the  potential  offerors. 
This  section  presents  a  modified  sample  briefing  used  during 
a  pre-proposal  conference  to  explain  the  SCE  process  and 
solicit  feedback  from  the  prospective  offerors.  Figure  5-5 
provides  a  title  slide  and  agenda.  The  presentation  consists  of 
the  following  parts  to  guide  the  interaction  with  the 
prospective  offerors; 

•  What  is  SCE,  What  are  the  activities  that  usually  take  place 
(Figure  5-6). 

•  The  SCE  team  process  (Figxore  5-6). 

•  A  description  of  validation  procedures  (Figure  5-6). 

•  A  sample  of  the  documentation  that  may  be  looked  at  by 
the  SCE  team  (Figure  5-7). 

•  A  sample  site  visit  schedule,  (Figure  5-8). 

•  The  CMM  and  KPAs  against  which  the  team  will  evaluate 
the  offerors  (Figure  5-9). 

•  A  sample  of  the  findings  the  SCE  team  will  produce  before 
leaving  the  site  (Figure  5-10). 
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USE  OF 

SOFTWARE  CAPABILITY  EVALUATION 
(SCE) 

IN  THE 

XYZ  CONTRACT 


Figure  5-5  Pre-proposal  Conference  Title  and  Agenda 
Slides 


SCE  description 
SCE  team  activities 
Key  points  about  the  site  visit 
Criteha  lor  validating  practices 
Sample  of  documents  reviewed 
Example  of  typical  site  visit  schedule 
Key  Process  Areas  (KPAs)  investigated 
Example  detailed  summary  findings 


Figure  5-6  shows  the  slides  used  to  describe  the  SCE  Method 
and  give  the  offeror  a  feel  for  the  site  visit  activities.  The  SCE 
Method  is  relatively  new  in  practice  and  there  are  offeror 
locations  that  have  not  experienced  an  SCF.  For  this  reason,  it 
is  especially  important  to  take  the  time  to  answer  any  SCE- 
related  questions  in  an  open  forum  such  as  an  offerors' 
conference.  Emphasis  should  be  placed  on  how  the  interviews 
will  be  conducted  (e.g.,  remain  confidential,  one  person 
interviewed  at  a  time). 

One  of  the  key  differences  between  the  SCE  and  SPA  methods 
is  the  examination  of  documentation  to  validate  the  processes. 
Keep  in  mind  when  presenting  the  slide  on  validating 
practices  that  validation  occurs  after  examining  the  audit  trail 
information.  This  audit  trail  information  reveals  how  certain 
processes  or  practices  are  implemented.  The  documents 
which  may  help  describe  these  practices  are  listed  in  Figure  5* 
7. 


CMU/SEl  93-TR-18 


Chapter  5;  Incorporating  SCE  into  the  Relevant  Acquisition  Documentation 


Figure  5-6  Pre-proposal  Conference  SCE  Method  Slides 
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Figure  5-7  provides  a  broad  list  of  the  documentation  that  the 
SCE  team  may  examine  depending  upon  the  course  of  the 
interviews.  This  list  of  documents  is  a  guide  to  better  prepare 
the  offerors.  Many  more  documents  may  actually  be  reviewed 
during  the  site  visit. 


SAMPLE  OP  DOCUMENTS  REVIEWED 

•  Current  organization  chart. 

•  Corporate  policy/procedures  on  software  man¬ 
agement. 

•  Project  policy/procedure  on  software  manage¬ 
ment. 

•  Software  development  plans. 

•  Software  configuration  management  plans. 

•  Software  quality  assurance  plans. 

«  Training  plans. 

•  Current  metrics  (timing  and  sizing,  etc.). 

•  Inplementation  procedures  ano  company  stan¬ 
dards. 


Figure  5-7  Pre-proposal  Conference  Documentation  Slide 
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The  site-visit  schedule  (Figure  5-8)  should  be  mentioned 
during  the  briefing  in  order  to  prepare  offerors  for  what  will 
occur  during  the  three-day  site  visit.  It  should  be  emphasized 
that  the  SCE  team  will  do  everything  possible  to  minimize  the 
impact  on  the  offeror's  software  development  organization. 


SAMPLE  SITE  VISIT  SCHEDULE 

DAYO 

0800-1330  Travel  for  SCE  team 
1 330-1 800  Prepare  tor  site  visit 

DAY1 

0800-0830  Contractor  welcome/introduction  and  SCE  team’s  orientation  briefing 
0830-1 030  Contractor  software  presentation/contractor  understanding  caucus 
1 030-1230  Initial  review  of  organizational  documents 
1 230-1 330  Lunch 

1 330-1 430  SCE  team  interviews  -  senior  managers 
1 430-1 600  SCE  team  interviews  •  program  managers 
1 600-1 730  SCE  team  interviews  -  software  managers 
1 730-1 800  SCE  team  caucus  and  requests  documents 

DAY  2 

0800-1 000  SCE  team  interviews  continued  with  program  managers,  software 
managers 

1000-1200  SCE  team  interviews  -  managers  (engineering.  SQA,  SCM,  test,  SEPG) 
1200-1300  Lunch 

1300-1400  SCE  team  cauc  US,  request  and  review  documents 


DAY  3 

0800-1000 

1000-1200 

1200-1300 

1300-1500 


SCE  team  interviews  -  key  personnel  (may  include  engineering  staff) 
Review  additional  documentation 
Lunch 

Additional  documentation  review  and/or  additional  SCE  team  intervievra  - 
Consolidation  interviews  with  managers,  engineers,  and  staff 
1 500-1 800  SCE  team  caucus  and  preparation  of  findings 


DAY  4 

0800-0900 

0900-1000 

1000-1100 

1100-1730 


Final  preparation  of  findings 
Exit  meeting  with  offeror 
SCE  team  caucus  and  wrap-up 
Travel  for  SCE  team 


mm, 


Figure  5-8  Pre-proposal  Conference  Site  Visit  Schedule 
Slide 
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If  the  team  is  planning  to  visit  the  engineering  floor  (software 
engineer's  work  areas)  at  each  site,  the  team  should  explain 
how  this  will  be  done,  should  explain  that  the  SCE  team  will 
do  this  activity  at  a  mutually  agreeable  time  during  the  site 
visit,  and  should  estimate  the  duration  of  the  floor  visit 
(normally  NTE  4  hours  per  project  visited). 

The  CMM  should  also  be  explained  so  that  the  basis  of  the 
SCE  is  clear  (Figure  5-9).  The  offerors  need  to  understand 
what  KPAs  are  going  to  form  the  basis  of  the  specific  criterion, 
or  be  considered  under  performance  risk,  incorporating  the 
SCE  findings  into  the  source  selection  decision.  Depending 
upon  the  offeror  pool  and  their  familiarity  with  the  CMM, 
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additional  slides  may  be  needed  to  explain  the  subprocess 
areas  or  key  practices  for  each  KPA  the  team  will  be 
examining  during  the  SCE. 


Level 

Characterietic 

Key  Process  Area 

Optimiaing  (S) 

•  Continuous  process 
improvement  capability 

•  Process  change  management 

•  Technology  innovation 

•  Defect  prevention 

Managed  (4) 

•  Product  quality  planning 
and  tracking  o1  mea¬ 
sured  software  process 

•  Process  measurement  and 
analysis 

•  Quality  management 

Defined  (3) 

•  Life  cycle  process 
defined  and  institutional¬ 
ized  to  provide  product 
quality  control 

•  Peer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 

•  Integrated  software  manage¬ 
ment 

•  Training  program 

•  Organization  process  defmi- 
tbn 

•  Organization  process  focus 

Repeatable  (2) 

•  Management  oversight 
and  tracking  of  project 

•  Stable  planning 

•  Software  configuration 
management 

•  Software  quality  assurance 

•  Software  subcontract  manage¬ 
ment 

•  Software  project  tracking  and 
oversight 

•  Software  project  planning 

•  Requirements  management 

Initial  (1) 

•  Ad  hoc  (unpredictable, 
chaotic) 

•  “People’ 

Figure  5-9  Pre-proposai  Conference  CMM  Slide 

The  last  slide  (Figure  5-10)  in  the  SCE  portion  of  the  offerors' 
conference  briefing  explain  to  the  offerors  what  the  results  of 
the  site  visit  will  look  like  and  how  they  will  be  presented  to 
the  SSEB.  It  is  not  possible  to  give  specifics  on  the  evaluation 
standard  or  the  percentage  that  the  technical  item  Software 
Engineering  Capability  contributes  to  the  source  selection 
decision;  however,  the  order  of  importance  SCE  has  when 
used  as  an  item  under  the  technical  area  can  be  emphasized. 
This  is  an  opportunity  to  reinforce  the  point  that  all  findings 


CMU/SEI-93-TR-18 


5-99 


Chapter  5:  incorpomtirrg  SCE  into  the  Relevant  >^uisftion  Documentation 


are  made  through  corisensus  by  the  team.  Sample  findings 
charts  should  reflect  the  key  process  areas  that  are  going  to  be 
evaluated  for  the  acquisition. 


SAMPLE  DETAILED  FINDING 
SQAKPA 

Strengths 

•  Independent  reporting  chain 

•  Highly  visible 

•  Insuring  software  engineering  standards 
compliance 

•  Management  commitment  -  strong  staff 

Weaknesses 

•  Inconsistent  auditing 

•  Ineffective  use  of  resources 

Improvement  Activities 

•  Establishing  procedures  for  consistent 
auditing 


EXAMPLE  SUMMARY  FINCMNG 
Strong 

•  Software  Quality  Assurance  (SQA) 

Acceptable 

•  Project  Planning 

•  Software  Configuration  Management 

•  Standards  and  Procedures 


Weak 

•  Project  Management 

•  Software  Engineering  Process  Group 

•  Training 

•  Peer  Reviews 


Figure  5-10  Offerors’  Conference  Sample  Findings  Slides 


Placing  SCE  in 
the  Request  for 
Proposal 


This  section  contains  the  information  needed  to  incorporate 
SCE  into  the  Request  For  Proposal  (RFP).  The  RFP  is  the 
document  used  by  the  government  to  explain  to  offerors 

•  The  government's  requirements. 

•  Evaluation  criteria. 

•  The  mechanisms  that  will  be  employed  in  making  the 
source  selection  decision. 

•  How  to  propose  for  the  contract. 

The  examples  provide  RFP  language  segments  that  eliminate 
all  references  to  maturity  levels. 
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General  Language 
for  the  Request  for 
Proposal 


All  of  the  examples  in  this  section  revolve  around  Software 
Engineering  Capability  being  used  as  a  specific  criterion  in 
the  Technical  area  of  the  proposal.  If  the  SCE  findings  are 
planned  to  be  used  as  a  consideration  under  performance 
risk,  these  examples  can  be  easily  tailored  to  meet  such  usage. 

Regardless  of  how  SCE  is  to  be  used  in  making  the  source 
selection  decision,  the  description  of  its  use  is  found  in  Section 
L  (Instructions,  Conditions,  and  Notices  to  Offerors)  and  M 
(Evaluation  Factors  for  Award)  of  the  RFP.  The  example 
shown  in  Figure  5-11  closely  mirrors  an  actual  description  of 
SCE  use  foimd  in  Section  M  of  an  RFP.  The  example  begins  by 
identifying  Software  Engineering  Capability  as  an  item  under 
the  technical  area  (specific  criterion). 

For  this  example,  there  were  two  areas  of  evaluation:  1) 
Technical  and  2)  Cost  The  specific  reference  to  SCE  in  the  RFP 
begins  by  describing  in  general  terms 

•  What  will  be  evaluated  in  the  technical  area  (process  capa¬ 
bility)  and  the  importance  placed  on  each. 

•  What  the  technical  basis  of  the  evaluation  is  (the  CMM 
KPAs) 

•  What  the  results  of  the  evaluation  will  be  (identify 
strengths,  weaknesses,  and  risk  which  will  also  consider 
improvement  activities  by  KPA) 

•  How  the  government  will  conduct  the  evaluation  (select 
the  projects  to  be  reviewed,  conduct  interviews,  and  re¬ 
view  documentation). 


CMU/SEI-93-TR-18 


5-101 


Chapter  5:  Incorporating  SCE  into  the  Relevant  ^:quisition  Documentation 


.  # 


B.  Evaluation  Criteria 

1.0  Introduction 

2.0  Basis  for  Contract  Award 

3.0  General  Considerations 

4.0  Assessment  Criteria 

4. 1  Specific  Criteria 

4.1.1  Technical  Area 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (SEC,  TECH  approach  and 
Management)  represented  here  in  descending  order  of  importance; 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute  (SEI)-developed  Software  Capability  Evaluation  (SCE).  The 
government  will  determine  the  software  process  capability  by  investigating  the  Key 
Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Capability  Maturity  Model  for 
Software  {CMUJSEl-S'i -1^-24,  Number  ADA  24603,  August  1991).  The  report  contains  a 
description  of  the  Capability  Maturity  Model  (CMM)  and  the  SEI-defined  maturity  levels. 
The  government  will  perform  an  SCE  of  each  offeror  remaining  in  the  competitive  range  by 
reviewing  current  projects  at  the  site  proposing  on  this  contract  and  comparing  methods/ 
processes  used  on  those  project  in  the  written  proposal/SPIP. 

The  evaluation  will  result  in  an  organizational  composite,  substantiated  through  individual 
inten/iews  and  reviews  of  documentation,  of  the  offeror’s  software  process  activKies  on  the 
government-selected  projects.  A  risk  usessment  to  compare  proposed  practices  to 
current,  validated  practices  may  be  performed.  The  evaluation  team  will  determine  findings 
of  the  offeror's  strengths,  weaknesses,  and  improvement  activities  in  all  KPAs  through  the 
Defined  maturity  level.  Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will 
be  provided  in  writing  during  subsequent  discussions.  The  offeror  will  be  allowed  to 
respond  to  the  findings  with  a  limited  number  of  pages  of  text.  The  on-site  evaluators  may 
be  separate  and  distinct  from  the  proposal  evaluation  team  and  may  include  a  government 
contracting  representative.  All  evaluators  have  been  trained  in  the  SCE  Method. 

4.2  Cost/Price  Area... 


Eliminating 
References  to 
Maturity  Levels  in 
the  RFP 


*  ' 


Figure  5-1 1  General  RFP  Language  To  Include  SCE  (RFP 
Section  M) 

While  Figure  5-11  treats  the  maturity  level  as  a  basis  for 
evaluation  rather  than  a  requirement,  it  is  recommended  that 
references  to  maturity  level  be  eliminated  to  the  greatest 
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References  to  SCE 
in  the  Instructions 
for  Proposal 
Preparation 


extent  possible.  One  way  to  do  this  is  by  listing  the  actual 
KPAs  and  subprocess  areas  or  key  practices  to  be  evaluated 
directly  in  the  RFP  rather  than  simply  referencing  the  KPAs 
found  in  the  CMM.  Eliminating  references  to  maturity  levels 
allows  the  acquirer  to  establish  a  target  process  capability 
using  the  CMM  KPAs  and  also  to  be  specific  as  to  the  basis  of 
evaluation.  It  is  the  findings  against  the  KPAs  that  are  most 
important  to  the  acquirer  in  determining  specific  risk  in  an 
offeror,  not  the  maturity  level. 

The  Instructions  For  Proposal  Preparation  (IFPP)  portion  of 
the  RFP  informs  the  offerors  how  to  prepare  their  proposals 
and  comply  with  the  source  selection  process.  The  portion  of 
the  IFPP  that  addresses  SCE  is  divided  into  five  distinct 
pieces: 

1.  Organizing  the  SCE -related  information  into  a  separate 
appendix. 

2.  Completing  the  Assessment  Recording  Forms  (Figure  5- 

12). 

3.  Completing  the  Project  Profiles  (Figure  5-13). 

4.  Submitting  the  Software  Process  Improvement  Plan  (SPIP) 
(Figure  5-14). 

5.  General  SCE  instructions  (Figiure  5-15). 

Organizing  SCE-related  Information  into  an  Appendix 
Each  acquisition  must  determine  the  best  way  for  their 
program  to  instruct  offerors  to  organize  their  proposals.  The 
goal  is  to  ease  proposal  evaluations  and  obtain  concise 
information  wanted,  and  not  to  impose  a  burden  on  the 
offerors.  Work  with  your  PCO  to  determine  what  is  best  for 
your  individual  acquisition  and  program.  If  it  is  desired  that 
the  SCE  information  be  excluded  from  the  technical  page 
limit,  you  may  want  offerors  to  place  this  information  in  a 
separate  appendix. 
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Completing  Assessment  Recording  Forms 
One  of  the  more  important  SCE-related  items  in  the  IFPP  is 
the  language  shown  in  Figure  5-12  describing  how  the 
Assessment  Recording  Forms  are  to  be  completed.  The  offeror 
is  told  to  select  eight  on-going  development  efforts  that  best 
correlate  to  the  future  work  imder  the  government's 
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proposed  contract,  using  characteristics  such  as  application 
domain,  software  size,  development  language,  etc.  to  make 
the  best  matches. 


roposal 


3.1  General  Inatructione  The  Technical  Proposal  shall  consist  of  the  Executive  Summary  and 
V  number  of  separate  sections... 

3.2  Executive  Summary... 

3.3  Specific  Inatructione... 

3.3.1  Section  I  -  Software  Engineering  Capability.  The  offeror  will  provide  the  following 

information  to  assist  the  government's  preparation  for  the  Software  Capability  Evaluation  of 
each  offeror: 

a.  The  offeror  wilt  complete  the  SEt  Maturity  Questionnaire  (MQ)  (see  A  Method  lor 
Assessing  the  Software  Engineering  Capability  of  Contractors,  CMU/SEI-87-TR-23,  OTIC 
Number  ADA  1 87320,  September  1987)  using  the  Assessment  Recording  Form  for  eight 
current  projects  at  the  site  proposing  on  this  solicitation  (a  project  is  acceptable  only  if  it 
has  been  completed  in  the  last  year).  The  offeror  should  select  those  projects  that  best 
match  the  engineering  requirements  of  this  contract.  For  offerors  with  fewer  than  eight 

■  current  projects  at  the  proposing  site,  submit  MQ  responses  for  as  many  projects  as  are 

available.  For  each  “yes"  response,  please  note  on  the  comment  line  the  mechanism  or 
documentation  justifying  the  response.  The  MQ  can  be  found  in  Atch  XXX  of  the  IFPP.  The 
completed  Assessment  Recording  Forms  will  be  submitted  with  the  proposal.  For  all 
responses,  please  note  at  the  start  of  the  comment  line  the  degree  of  implementation  of 
each  practice  using  a  letter  identifier  from  the  following  legend; 

A  ■  Not  implemented  at  this  time. 

B  -  Not  implemented  at  this  time,  but  desired. 

C  •  Currently  planning  to  implement.  See  improvement  plan. 

0  -  In  the  process  of  inplementing. 

E  -  Implemented  with  less  than  a  year’s  experience. 

F  -  Implemented  on  a  project-by-project  basis. 

G  -  Implemented  organizationally. 

H  -  Not  appropriate  for  our  organization. 


■ 


Figure  5-12  Instructions  For  Completing  Assessment  Re¬ 
cording  Forms 


Using  the  legend  in  Figure  5-12,  the  offeror  must  characterize 
the  state  of  institutionalization  regarding  each  practice.  To 
verify  each  response,  the  offeror  must  cite  documentation  that 
defines  each  practice  it  has  characterized.  By  getting  this 
information  from  offerors,  the  SCE  team  will  know  more 
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about  what  to  look  for  to  verify  a  particular  software  practice, 
and  the  offeror's  view  of  the  extent  to  which  a  practice  is 
implemented,  which  will  help  focus  on-site  activities  (e.g., 
interviews). 

The  SCE  team  wants  to  identify  and  separate  software 
practices  that  are  institutionalized  (implemented 
organizationally)  from  those  that  are  implemented  only  on  a 
project.  The  last  reference  in  Figure  5-12  is  to  the  maturity 
questionnaires  the  offeror  must  complete  for  each  project 
submitted  in  order  to  satisfy  this  request.  The  questionnaire 
should  be  attached  to  the  RFP  so  that  there  is  no  confusion 
over  which  questionnaire  the  offeror  is  required  to  complete 
{A  Method  for  Assessing  the  Software  Engineering  Capabilty  of 
Contractors  [Humphrey  87b). 

Completing  the  Project  Profiles 

Figure  5-13  directs  the  offeror  to  complete  a  Project  Profile  for 
each  of  the  projects  selected  on  the  Assessment  Recording 
Form.  The  project  profile  provides  information  such  as:  size 
of  the  organization,  application  area,  development  language, 
type  of  system,  location  of  development,  and  the  program's 
current  phase(s)  of  development.  This  information  helps  the 
government  in  selecting  projects  for  evaluation.  A  Project 
Profile  should  also  be  included  as  an  attachment  to  the  IFPP. 


b.  For  each  project,  the  offeror  will  complete  a  Project  Profile,  attach  it  to  the  respective 
Assessment  Recording  Form,  and  submit  it  with  the  proposal.  The  Project  Profile  template 
can  also  be  found  in  Atch  XXX  of  the  IFPP.  This  document  shall  be  no  greater 
than  one  page  per  project. 


Figure  5-13  Instructions  For  Completing  Project  Profiles 

Preparing  the  Software  Process  Improvement  Plan  (SPIP) 

Figure  5-14  addresses  the  SPIP  the  offerors  submit  with  their 
proposals.  In  the  example  provided  the  offeror  could  not 
exceed  15  pages  of  text.  This  was  an  arbitrary  limit  intended 
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to  minimize  the  effort  required  by  the  offerors  and  the 
govermnent  during  the  source  selection  process.  The 
government  did  not  want  the  offerors  to  develop  an  SPIP  for 
the  acquisition;  rather,  they  wanted  to  see  plans  that  were 
already  in  place. 


3.31  (continued) 


c.  The  offeror  will  submit  the  proposing  site’s  Software  Process  Improvement  Plan  (SPIP). 
in  the  format  of  their  choosing,  with  the  proposal.  The  document  will  be  no  more  than  1 5 
pages.  The  SPIP  should  communicate  the  offeror’s  current  software  process  capability  as 
well  as  their  desired  maturity  level,  specific  planned  improvements,  dedicated  resources, 
effort  estimates,  and  a  time  phasing  of  those  improvements  to  bring  the  offeror's  software 
process  capability  to  the  organization's  desired  maturity  level. 


;  ,*4.  ' 

'  <  'A  i ''' 

Figure  5-14  instructions  For  Submitting  the  Software  Pro¬ 
cess  Improvement  Plan  (SPIP) 

General  SCE  Instructions 

The  last  set  of  instructions  for  the  IFPP,  found  in  Figure  5-15 
informs  the  offeror  of  various  SCE-related  details  that  will 
facilitate  a  smoothly  run  SCE  with  minimal  impact  on  the 
offeror's  organization. 

•  An  Offeror  Point  of  Contact  is  needed  so  that  the  SCE  team 
leader  may  coordiiwte  all  activities,  both  before  and  dur¬ 
ing  the  SCE.  Note  that  the  offeror  will  be  notified  five 
working  days  in  advance  of  the  site  visit  which  projects 
will  be  evaluated.  There  are  two  reasons  for  this.  First,  this 
will  give  all  the  offerors  the  same  number  of  days  to  pre¬ 
pare  for  the  SCE.  Second,  because  many  organizations  go 
to  great  lengths  to  prepare  for  an  SCE,  giving  five  working 
days'  notice  will  limit  them  from  expending  valuable  re¬ 
sources  and  time  on  activities  that  will  have  little  or  no  im¬ 
pact  on  the  SCE  findings. 
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3.31  (continued) 


d.  After  the  proposal  is  received,  the  government  wilt  coordinate  a  site  visit  with  those 
offerors  remaining  in  the  competitive  range  to  conduct  the  Software  Capability  Evaluation 
(SCE)  at  the  offeror's  location.  The  offeror  will  provide,  with  your  proposal,  a  point  of 
contact  and  phone  number  at  the  offeror's  site  for  the  SCE  team  leader  to  coordinate  all 
SCE  activities.  The  government  will  also  communicate  details  about  the  site  visit  during 
the  coordination  process.  The  offeror  will  bo  notified  of  the  projects  to  be  examined 
approximately  five  working  days  prior  to  the  site  visit.  The  site  visit  dates  selected  by  the 
government  are  not  open  for  discussion. 

e.  If  a  site  visit  is  conducted  with  your  firm,  the  SCE  team  will  need  a  closed  meeting  room 
capable  of  accommodating  at  least  eight  people.  The  offeror  should  have  a  copy  of  the 
organization's  software  standards,  procedures  and/or  operating  instructions,  and 
organizational  charts  for  the  projects  being  reviewed  in  the  meeting  room  when  the  SCE 
team  arrives.  All  interviews  conducted  as  part  of  the  SCE  will  be  done  in  private,  one 
individual  at  a  time. 

f.  The  Assessment  Recording  Forms,  Project  Profiles,  and  Software  Process  Improvement 
Plans  will  not  be  included  in  the  page  count  limitations  for  the  proposal. 


Figure  5-15  Instructions  For  Site  Visit  Coordination 


•  Facilities  and  Information.  The  items  needed  by  the  team  at 
the  site  visit  are  mentioned  in  this  section.  This  informa¬ 
tion  needs  to  be  provided  here  to  set  expectations  and  en¬ 
sure  that  the  offeror  is  reasonably  well  prepared.  Note  that 
private  interview  notes  will  always  remain,  source  selec¬ 
tion  sensitive.  The  SCE  team  must  maintain  the  confiden¬ 
tiality  of  interviews  or  the  entire  SCE  process  could  be  un¬ 
dermined.  All  data  collected  during  the  site  visit  will  be¬ 
come  part  of  the  source  selection  file  and  will  be 
maintained  on  all  offerors  the  contract  is  closed  out. 


•  Offeror  Exit  Briefing .  The  PCO  will  be  the  final  arbiter  in 
determining  how  the  findings  will  be  provid#‘d  to  the  off¬ 
erors.  However,  any  outbriefing  must  advise  the  offeror 
that  this  may  not  completely  resolve  all  issues  regarding 
the  SCE.  It  is  important  for  all  the  offerors  to  realize  that 
they  have  the  right  and  must  specifically  request  a  debrief¬ 
ing  of  the  SCE  findings.  Debriefing  the  findings  achieves 


5-108 


CMU/SEI-93-TR-18 


Chapter  5.  Incorporating  SCE  into  the  Relevant  Acquisition  Documentation 


Placing  SCE  in 
the  Evaluation 
Plan 


two  important  goals.  First,  in  a  Total  Quality  Management 
(TQM)  approach,  the  government  desires  buy-in  from  the 
offerors  regarding  the  results,  and  is  seeking  to  motivate 
the  offerors  to  improve  their  capability.  Second,  the  gov¬ 
ernment  has  the  opportunity  for  direct  feedback  regarding 
the  conduct  of  the  SCE  from  the  offeror's  perspective.  This 
feedback  can  be  used  to  refine  the  procedures  and  instruc¬ 
tions  for  future  acquisitions. 

•  Page  Limitations.  In  most  RFPs,  there  is  a  limit  to  the  num¬ 
ber  of  pages  an  offeror  may  use  in  the  preparation  of  their 
proposal.  The  example  provided  here  had  such  a  require¬ 
ment.  Corrsequently,  when  the  IFPP  required  submittal  of 
Assessment  Recording  Forms,  Project  Profiles,  and  an 
SPIP,  these  document  pages  were  excluded  from  the  pro¬ 
posal  page  count  to  ensure  they  did  not  detract  from  the 
technical  content  of  the  proposal  subject  to  the  page  limi¬ 
tations.  This  is  an  administrative  detail  which  will  allow 
page  counts  to  be  focused  on  the  technical  approach. 

This  section  presented  the  essential  elements  needed  to 
accommodate  SCE  in  an  RFP.  These  references  should  be 
tailored  by  the  orgaruzation  to  meet  the  specific  needs  of  the 
acquisition.  The  references  presented  can  be  changed  to 
accommodate  the  usage  of  the  SCE  findings  as  a 
Consideration  under  performance  risk  or  a  variation  of  the 
Specific  Criterion  example  presented  here. 


This  section  provides  a  sample  of  the  type  of  information 
needed  to  incorporate  SCE  into  the  Evaluation  Plan,  and  to 
assist  in  the  preparation  of  an  evaluation  standard  for  SCE 
findings. 

Evaluation  standards  must  be  completed  before  RFP  release. 
Evaluation  standards  are  written  in  a  detailed  manner  to 
promote  competition  and  enhance  the  discrimination 
between  the  offerors. 
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It  is  imperative  that  the  SCE  team  leader  be  a  member  or 
advisor  to  the  Source  Selection  Evaluation  Board  (SSEB)  to 
work  with  SSEB  members  to  apply  the  evaluation  standard  to 
the  findings  from  each  of  the  offerors. 

The  example  presented  in  Figure  5-16  is  a  sample  evaluation 
standard.  A  detailed  evaluation  standard  is  written 
describing  the  requirements  for  the  respective  acquisition. 
This  implies  that  if  the  requirement  is  met,  the  standard  is 
met,  and  the  offeror  is  scored  Green.  If  the  standard  is 
exceeded,  the  offeror  is  scored  Blue.  If  the  requirement  is  not 
met,  depending  on  how  near  it  is  to  being  met,  the  offeror  is 
scored  as  Yellow.  A  Red  score  denotes  serious  deficiencies 
(failure  to  meet  requirements).  Actual  application  of  the  color 
ratings  to  a  standard  should  be  correlated  with  the 
appropriate  regulations  and  procurement  policies  affecting 
your  acquisition. 
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lOESCRIPTION:  Software  Engineering  Capability— The  government  will  evaluate  the  offeror's 
|organizational  software  process  capability  by; 

•  Performing  a  Software  Capability  Evaluation  (SCE). 

•  Evaluating  the  offeror's  program  for  software  process  improvement. 

•  Evaluating  the  extent  to  which  the  offeror's  software  process  supports  the  goals,  objectives,  and  require-j 
ments  of  the  solicKation. 

STANDARD-  The  standard  is  met  when  the  offeror  presents  s  sound,  compliant  approach  and: 

1 .  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  each  of  the  following  key 
process  areas; 

•  Software  Configuration  Management 

•  Sc^are  Quality  Assurance 

•  Software  Subcontract  management 

•  Software  project  tracking  and  oversight 

•  Software  project  planning 

•  Requirements  Management 

|2.  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  at  least  one  of  the  following  j 
software  process  areas: 

•  Peer  Reviews 

•  Intergroup  coordination 

•  Software  product  engineering 
»  Integrated  software  management 
»  Training  program 

•  Organization  process  definition 

•  Organization  process  focus 

|3.  The  Software  Process  Irr^roverTtent  Plan  submitted  with  the  offeror’s  proposal  portrays  the  offeror's 
current  process  capability  realistically  and  presents  a  realistic  plan  for  process  improvement.  The 
offeror's  plan  is  consistent  with  the  SCE  findings.  The  SPIP  outlines  the  offeror's  plan  to  achieve 
higher  maturity  levels  and  demonstrates  that  the  offeror  understands  software  process  improvement, 
both  technically  and  in  tha  effort  required  to  increase  and  sustain  higher  maturity  levels. 


Figure  5*16  Streamlined  SCE  Evaluation  Standard 

This  section  presented  an  example  of  an  evaluation  standard 
which  de-emphasizes  maturity  levels  while  keeping  with  the 
spirit  of  the  CMM.  Trained  SCE  users  are  able  to  take  this 
example  and  tailor  it  to  meet  the  specific  needs  of  their 
acquisition.  Thus,  SCE  can  contribute  effectively  to  the  source 
selection  decision.  The  findings,  provided  to  the  SSEB  by  the 
SCE  team,  are  a  snapshot  of  process  capability  for  a  specific 
site  at  a  particular  point  in  time.  The  way  those  findings  are 
used  by  the  acquisition  organization  can  be  modified  through 
the  design  of  the  evaluation  standard. 
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Appendix  A  SCE  Participants  in  a  Source 

Selection 

Figure  A-1  shows  how  a  source  selection  organization  may  be 
organized  with  the  incorporation  of  SCE. 


Figure  A-1  Sample  Source  Selection  Organization 

Note  that  in  this  example  the  cost  team  and  contract  definition 
team  are  separate.  They  would  be  combined  when  the  SSET 
structure  is  used. 

The  following  are  the  principal  source  selection  players  who 
influence  the  SCE  implementation  decision,  whether  it  will  be 
used  and  to  what  extent  the  SCE  findings  will  play  a  role  in 
the  source  selection  decision. 
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Source  Selection  Authority  (SSA):  As  the  individual 
ultimately  responsible  for  the  conduct  of  the  source  selection 
organization,  the  SSA  will  be  the  final  arbiter  on  the  use  of 
SCE  and  will  approve  how  the  findings  will  influence  the 
source  selection  decision  and  how  they  will  be  disclosed  to 
offerors. 

Source  Selection  Advisory  Council  (SSAC):  The  SSAC  is 
charged  with  collecting  and  analyzing  the  SSEB's  evaluations 
of  each  offeror.  This  is  the  only  group  permitted  to  compare 
the  strengths  and  weaknesses  of  the  offerors  against  one 
another.  The  SSAC  may  recommend  to  the  SSA  how  the  SCE 
findings  will  be  incorporated  into  the  source  selection 
decision  at  the  pre-RFP  release  briefing. 

SSAC  Chairperson:  This  individual  is  usually  the  facilitator 
of  the  entire  process  and  acts  as  the  SSA's  eyes  and  ears 
during  the  course  of  the  source  selection.  This  person  is 
normally  the  2-LTR  deputy,  or  at  least  in  the  chain-of- 
command  above  the  program  manager.  This  individual  will 
coordmate  all  activities  with  the  SSA  and  facilitate  the 
consensus-building  process  from  within  the  SSAC. 

Performance  Risk  Analysis  Group  (FRAG):  The  FRAG 
collects  data  on  each  offeror's  past  and  present  performance 
on  other,  similar  product  acquisitions  and  from  this  data 
determines  relevance  of  effort  and  assesses  the  performance 
risks  posed  in  the  offeror's  ability  to  perform  if  the  offeror  is 
selected.  SCE  findings  can  be  factored  into  this  performance 
risk  assessment.  The  FRAG  does  more  than  assess  past 
performance.  It  may  also  assess  such  things  as  manufacturing 
capability,  plant  capacity,  or  an  offeror's  familiarity  with  the 
type  of  work  required  under  the  instant  solicitation,  in  an 
attempt  to  analyze  an  offeror's  performance  risk.  The  FRAG 
normally  reports  directly  to  the  SSA  or  the  SSAC.  The  FRAG 
role  is  fully  defined  in  the  AFMC  FAR  supplement  to  AFR  70- 
15  and  AFR  70-30.  When  this  is  done,  the  FRAG  needs  to  be 
well-educated  on  what  SCE  is  and  what  it  can  provide  so  that 
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the  right  iitformation  is  passed  to  the  SSAC.  In  these  instances, 
it  is  strongly  encouraged  that  at  least  one  member  of  the 
PRAG  be  an  SCE-trained  government  individual. 

Source  Selection  Evaluation  Board  (SSEB):  This  is  the 
government  group  that  evaluates  the  offerors'  proposals 
against  evaluation  standards  and  section  M  of  the  RFP.  This 
group  develops  the  evaluation  standards  and  receives 
approval  to  use  them  from  the  SSAC  and  SSA  prior  to  the 
issuance  of  the  RFP.  The  SSEB  is  usually  organized  into 
technical  and  cost  teams  representing  each  of  the  evaluation 
areas  or  technical  disciplines  important  to  the  award  decision. 
Within  each  team,  each  individual  brings  the  technical  skills 
and  professional  experience  necessary  to  perform  a  fair  and 
effective  evaluation.  If  the  findings  of  an  SCE  are  being 
factored  into  the  source  selection  decision  as  an  Evaluation 
Criterion,  then  the  SCE  team  must  be  members  of  the  SSEB. 
The  SSEB  will  have  to  prepare,  prior  to  the  release  of  the  RFP, 
an  evaluation  standard  that  is  capable  of  capturing  the 
mearung  of  the  SCE  findings. 

SSEB/SSET  Chairperson;  The  SSEB/SSET  chairperson 
coordinates  all  activities  of  the  SSEB/SSET  related  to  the 
acquisitioa  The  chairperson  will  facilitate  the  incorporation 
of  SCE  into  the  source  selection  documentation  and  morutor 
the  various  evaluation  teams,  including  the  SCE  team. 

Program  Director  /  Manager  The  program  director /manager 
normally  orchestrates  the  acquisition  from  the  vantage  point 
of  the  SSEB/SSET  chairperson,  depending  upon  the  nature  of 
the  organization  and  dollar  size  and  criticality  of  the  contract. 
The  SCE  team  leader  will  have  to  work  with  the  program 
director  (if  he  or  she  is  part  of  the  source  selection,  otherwise 
it  would  be  the  SSEB/SSET  chairperson)  during  the  source 
selection  to  perform  SCEs  as  part  of  the  proposal  evaluation 
process  and  place  them  in  the  source  selection  plan. 

A  program  director  of  a  large  program  may  have  a  number  of 
contracts  under  the  umbrella  of  one  program  that  he  or  she  is 
responsible  for.  In  that  capacity,  a  program  manager  is 
typically  appointed  to  each  of  the  contracts  and  is  subordinate 
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to  the  program  director.  For  a  source  selection  under  these 
circumstances,  the  program  director  would  more  than  likely 
be  either  the  SS  AC  chairperson  or,  at  the  very  least,  a  member 
of  the  SSAC.  Ultimately,  however,  the  program  manager  is 
responsible  for  developing  the  acquisition  strategy,  getting 
the  requisite  approvals  to  pursue  the  strategy,  implementing 
the  acquisition  strategy,  and  executing  the  program  after 
contract  award. 

Procuring  Contracting  Officer  (PCO):  The  PCO  is 
responsible  for  all  commvmications  with  the  offerors,  and 
ensuring  that  the  entire  source  selection  process  is  consistent 
with  the  FAR  and  its  internal  department-  or  command- 
unique  regulations.  The  PCO  is  also  respor\sible  for  advising 
the  SSAC  on  the  interpretation  of  the  findings  to  ensure  a 
consistent  and  objective  award  decision.  Some  organizations 
have  had  some  of  their  procurement  persoimel  trained  in 
SCE. 

Legal  Advisor  /  Attorney:  AH  source  selectioixs  require  some 
degree  of  interaction  with  the  acquisition  organization's  legal 
staff.  For  this  reason  a  JA  individual  is  normally  a  member  of 
the  SSAC.  Some  organizatioas  have  had  legal  personnel 
trained  in  SCE.  This  training  is  extremely  important  because 
source  selections  are  implemented  many  different  ways 
throughout  the  government  and  acquisition  organizations 
may  employ  different  techniques  even  within  the  same 
government  agency. 

Software  Capability  Evaluation  (SCE)  Team:  This  is  the 
group  of  four  to  six  people  that  will  conduct  the  SCE  site  visits 
for  the  source  selection.  The  SCE  team  should  evaluate  each 
offeror's  Software  Process  Improvement  Plan  (SPIP)  and 
Software  Development  Plan  (SDP),  and  their  software  related 
portions  of  the  proposals.  'The  unique  circumstances  of  each 
contract  will  dictate  the  extent  of  the  contribution  the  SCE 
team  must  make  to  the  source  selection  (e.g.  RFP  familiarity, 
proposal  familiarity,  and  SSEB  participation). 
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SCE  Team  Leaden  This  individual  will  plan,  prepare,  and 
execute  the  SCE  for  an  acquisition.  The  SCE  team  leader's 
responsibilities  include  advising  the  SSEB/SSET  Chairperson 
regarding  the  specific  findings  of  the  SCE  team  and 
documenting  those  findings  in  writing,  may  be  required  to 
brief  SCE  portion  of  technical  area.  Within  the  constraints  of 
availability,  only  personnel  with  prior  experience  as  an  SCE 
team  member  should  be  assigned  to  this  position. 

Contractor  SCE  Point  of  Contact:  This  is  the  offeror's  focal 
point  for  an  SCE  site  visit.  The  SCE  team  leader  will  work  with 
this  individual  to  minimize  the  impact  of  SCE  interviews  and 
documentation  gathering  on  the  evaluated  organization.  The 
interaction  will  begin  prior  to  the  site  visit  to  coordinate 
activities  and  request  documentation  or  organizational  charts. 
An  important  point  to  remember  is  that  all  discussions,  both 
planned  and  unplarmed,  after  the  issuance  of  the  RFP  must  be 
through  the  PCO. 
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AFSB  Air  Force  Science  Board 

AMC  Army  Materiel  Command 

AMIS  Acquisition  Management  Information  System 

BAFO  Best  and  Final  Offer 

C  AO  Contract  Admirustration  Office 

CBD  Commerce  Business  Daily 

CDR  Critical  Design  Review 

CMM  Capability  Maturity  Model 

CPAR  Contractor  Performance  Analysis  Report 

CPEP  Contractor  Performance  Analysis  Program 

CRs  Clarification  Requests 

CSC  Computer  Software  Component 

CSCI  Computer  Software  Configuration  Item 

CSU  Computer  Software  Unit 

DC  AA  Defense  Contracting  Audit  Agency 

DoD  Department  of  Defense 

DTIC  Defense  Technical  Information  Center 

Dem/Val  Demorrstration/ Validation 

DRs  Deficiency  Reports 

DSMC  Defense  Systems  Management  College 

HMD  Engineering  Manufacturing  Development 
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ESC  Electronic  Systen\s  Center  (formally  ESD) 

FAR  Federal  Acquisition  Regulation 

FCA  Functional  Configuration  Audit 

GAO  General  Accounting  Office 

IFPP  Instructions  for  Proposal  Preparation 

IRS  Interface  Requirements  Specification 

JPO  Joint  Program  Office 

JTIDS  Joint  Tactical  Information  Distribution  System 
KPA  Key  Process  Area 

KSLOC  Thousand  Source  Lines  of  Code 

LTR  Letter 

MMP/CR  Manufacturing  Management  Production/Capa¬ 
bility  Review 

MQ  Maturity  Questionnaire 

NAWC  Naval  Air  Warfare  Center 

NRAD  NCCOSC  (Naval  Command,  Control,  and  Ocean 
Surveillance  Center)  RDT&E  (Research  Develop¬ 
ment  Test  and  Engineering)  Division 

NTE  Not  to  Exceed 

PCA  Physical  Configuration  Audit 

PCO  Procuring  Contracting  Officer 

PDR  Preliminary  Design  Review 

PEO  Program  Executive  Officer 

PFN  Point  For  Negotiation 

PM  Program  Manager 
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PRAG 

Performance  Risk  Analysis  Group 

RAI 

Request  for  Additional  Information 

RFP 

Request  For  Proposal 

SCE 

Softw'are  Capability  Evaluation 

SCM 

Software  Configviration  Management 

SDD 

Software  Design  Document 

SDP 

Software  Development  Plan 

SDIO 

Space  Defense  Initiative  Office 

SDR 

System  Design  Review 

SEI 

Software  Engineering  Institute 

SEPG 

Software  Engineering  Process  Group 

SOW 

Statement  of  Work 

spn? 

Software  Process  Improvement  Plan 

SPA 

Software  Process  Assessment 

SQA 

Software  Quality  Assurance 

SRR 

System  Requirements  Review 

SRS 

Software  Requirements  Specification 

SSA 

Source  Selection  Authority 

SSAC 

Source  Selection  Advisory  Council 

SSDD 

System/Segment  Design  Document 

SSEB 

Source  Selection  Evaluation  Board 

SSET 

Source  Selection  Evaluation  Team 

SSEG 

Source  Selection  Evaluation  Guide 

SSP 

Source  Selection  Plan 
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USAF  United  States  Air  Force 
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Appendix  D  Detailed  SCE  implementation 

Checklist 

The  following  checklist  has  been  extracted  from  the  text  of  the 
document.  The  list  is  not  intended  to  cover  all  situations  or 
contingencies,  but  instead  serve  as  a  handy  reference  and 
guide  that  is  not  as  voluminous  as  the  implementation 
procedures  document  itself. 

Detailed  SCE  implementation  checklist: 

Acquisition  Start  □  Develop  initial  awareness  (SPO  and  PCO  organizations) 

□  Determine  Applicability  to  this  acquisition 

□  Review  policy 

□  Review  acquisition  strategy 

□  Determine  acquisition  needs 
O  Develop  SCE  implementation  recommendation 

□  Input  to  CBD 

□  Input  to  acquisition  strategy  document 
O  Input  to  Pre-proposal  Conference 

□  Input  to  source  selection  plan 

□  Champion  implementation  of  SCE  on  acquisition 

□  Obtain  commitment  to  use  SCE  (resources  and  affected 
organizations  support) 

□  Review  SCE  team  leader  and  team  member  qualification 
criteria 

□  Ensure  applicability  of  team  criteria  to  acquisition 

□  Applicable  acquisition  personnel  attend  SCE  Overview 

□  Prepare  candidate  SCE  team  member  list 

O  Obtain  commitment  from  candidate  team  member's 
organization 


Organize /Select 
SCE  Team 
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Prepare  Plan 
and  Schedule 


Execute  SCE 
Methodology 
Preparation 


O  Selected  team  familiarized  with  acqviisition  and  applicable 
policy 

□  Selected  team  members  attend  SCE  training  as  necessary 

□  Determine  SCE  placement  within  acquisition  structure 
(source  selection) 

□  Prepare  recommendation  as  to  how  SCE  findings  are  to  be 
integrated  with  acquisition  (source  selection)  SSEB(e.g. 
area,  item) 

□  Determine  disf)osition  of  SCE  data 

□  Debriefing  of  unsuccessful  contractors 

□  Estimate  number  of  contractors  and  potential  geographic 
sites  to  be  visited 

□  Estimate  time  required  and  costs  (manpower,  travel, 
support) 

□  Determine/schedule/implement  preliminary  SCE  tasks 

□  CBD  announcement  input  completed 

□  Pre-proposal  Conference  Briefing  prepared 

□  Acqixisition  Plan,  Source  Selection  Plan,  RFP  SCE 
language  insertion 

O  Request  completion  of  Maturity  Questioni\aire  and  project 
profiles 

□  Instructions  on  how  to  submit  material 

□  Prepare  Evaluation  Standards  for  transforming  SCE 
findings 

O  Develop  Product  Profile 

□  Determine  Target  Process  Capability  (TPC) 

□  Schedule  SCE  team  to  meet  and  execute  SCE  Method  pre- 
onsite  phase  preparation. 

O  Analyze  project  profiles 

□  Select  contractors  projects 


D-128 


CMU/SEI-93-TR-18 


Appendix  D:  Detailed  SCE  Implementation  Checklist 


Conduct  SCE 


Write /Submit 
Finai  Report  to 
Acquisition 
Organization 


□  Identify  critical  subprocess  for  all  contractors 

□  Determine  key  issues  for  individual  contractors 

O  Develop  initial  exploratory  interview  questions  and 
identify  irutial  set  of  documents  for  review 

□  Develop  and  notify  contractor  points  of  contact  regarding 
SCE  team  logistical  requirements  (minimum  one  week  in 
advance) 

□  Room,  table,  chairs,  documents,  preliminary  on-site  and 
interview  schedules 

□  Conduct  SCE  on-site  phase  by  overall  acquisition  schedule 

□  Conduct  in-briefing  with  on-site  contractor 

O  Analyze  orgaruzational  and  project  documentation 

□  Review  and  modify  agenda  and  schedule  as  necessary 

O  Conduct  exploratory  interviews 

□  Request  additional  documentation 

□  Validate  interview  responses 

□  Draft  preliminary  findings 

O  Validate  preliminary  findings 

□  Conduct  coi\solidation  interviews 

□  Validate  improvement  activities 

□  Develop  final  findings 

O  Conduct  exit  briefing/meeting  (as  prescribed  by 
Procuring  Contracting  Officer  (PCO) 

□  Document  conduct  and  rationale  for  SCE  and  its  findings 

O  Develop  lessons  learned/feedback  to  improve  SCE 
methodology 

□  Document  effort  and  resources  expended 
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Assist 
Acquisition 
Organization’s 
Use  of  SCE 
Findings 


Formal 

Feedback 


□  Consult  with  SSEB  and  SSAC  as  needed  (elaborate  on  SCE 
findings) 

□  Integrate  SCE  findings  into  source  selection 

□  Develop  and  deliver  final  SCE  results  briefing  to  SSEB  (if 
necessary) 

□  Assist  SSEB  in  preparing  and  deliv  ering  formal  SCE 
presentation  to  the  SSAC 

□  Conduct  SCE  findings  briefing  for  winning  contractor 

O  Conduct  SCE  findings  briefings  to  losing  contractors 

□  Dispose  of  SCE  data  (In  accordance  with  acquisition 
guidelines) 

□  Disband  SCE  team 
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